Thursday, June 4, 2015

using ed

Having a moment of spare time I decided to go back to my approach to 99 scala problems. When I started I have used pure scala scripts, so I needed eg. to call them like this:
scala -i p05.scala -i p04.scala -i p03.scala p06.scala
Now I've learnt that it may be easier to go on using
sbt console
However my code was not sbt-friendly, as it caused complaints regarding missing object or class definition. You cannot just define methods like in REPL. So I ended up with urgent neccessity to include my functions into some nice objects. As I am currently playing with rather small machine full IDE like IntelliJ is not an option. So basically I needed to use my command line to:
  • add object definition line before content
  • add closing brace after content
  • where the latter is trivial append to file, usually
    echo "}" >> P01.scala
    
    How to append a line of text at beggining of file? Enter ed.
    Ed is an editor with tradition, used to scarry out younger programmers in team. I have read a fantastic intro to ed not so long ago. This is the ed script for making the changes with line by line explanation what it is doing (note ed will not get this comment):
      
    1                                    //goto first line 
    i                                    //start inserting
    package konopski.skala.ninety.nine { //actual text
      object PP00 {                      //more text
    .                                    //single dote
                                         //ends writing
    $                                    //goto last line
    a                                    //append 
      }                                  //some  
    }                                    //text
    .                                    //ok, that's enough
    
    w                                    //wq seems familiar
    q                                    //have you left any program this way?
    
    After that it took one line in shell I use to complete the task:
      
    for p in p*.scala; cat edme | sed -e s/PP00/$p -e s/\.scala// | ed $p ; end
    
    Note I have used ed's famous cousin - sed to replace placeholder value PP00 to actual file name without extension. That's it, the massive work is done and I have not used copy/paste or even any vim macro. I must say it was amusing equally to playing with scala, however seems to be even more practical than next way to reverse a list ;>

    Wednesday, April 15, 2015

    jq one more time - see only what really matters

    Sometimes you can find lot of information noise. Say I want to see only document id and it's content. Check this:
      
     curl -XPOST "http://es1.grey:9200/settings/d/_search?limit=50" -d'
                             {"query": { "match_all": {}}}' | jq ".hits.hits[] | [ ._id,  ._source ]"
    

    Friday, January 23, 2015

    new favourite command

    A must have for console addicted git users:
    git checkout -
    
    This jumps back and forward between two last branches. Really awesome.

    Thursday, January 8, 2015

    more hints regarding Optionals

    Instead of
            Optional<Camera> camera = getCameraByMac(cameras, mac);        
            if (camera.isPresent()) {            
                return Optional.of( camera.get().getIp() );        
            } else {            
                return Optional.absent();        
            }
    
    
    use
      
       Optional<Camera> camera = getCameraByMac(cameras, mac);
       return camera.transform(getIpFunction);
    
        static final Function<Camera, String> getIpFunction = new Function<Camera, String>() {
            @Nullable @Override
            public String apply(Camera camera) {
                return camera.getIP();
            }
        };
    
    
    instead of
      
            List<Event> result = new LinkedList<>();
            for (Optional<Event> transformedEvent : someEvents) {
                if (transformedEvent.isPresent()) {
                    result.add(transformedEvent.get());
                }
            }
            return result;
    
    
    use
      
            return Optional.presentInstances(events);
    
    

    Thursday, December 18, 2014

    Using jq to talk with elasticsearch

    I've recently found a new tool, quite useful, if you often talk to json services. It's named jq, please do not confuse it with jQuery. Eg. let's take Elasticsearch. I want to know which JVM it runs within. So I need to ask like this:
    curl -XGET es1.grey:9200/_cluster/stats
    so here's the output:
    {"timestamp":1418913590962,"cluster_name":"gc-pb-grey","status":"green","indices":{"count":31,"shards":{"total":252,"primaries":126,"replication":1.0,"index":{"shards":{"min":4,"max":10,"avg":8.129032258064516},"primaries":{"min":2,"max":5,"avg":4.064516129032258},"replication":{"min":1.0,"max":1.0,"avg":1.0}}},"docs":{"count":162713289,"deleted":188051},"store":{"size":"44.2gb","size_in_bytes":47543046975,"throttle_time":"1.9d","throttle_time_in_millis":172301216},"fielddata":{"memory_size":"2.6gb","memory_size_in_bytes":2858909795,"evictions":0},"filter_cache":{"memory_size":"4.8gb","memory_size_in_bytes":5161692177,"evictions":504563},"id_cache":{"memory_size":"0b","memory_size_in_bytes":0},"completion":{"size":"0b","size_in_bytes":0},"segments":{"count":2000,"memory":"2.9gb","memory_in_bytes":3169080575}},"nodes":{"count":{"total":8,"master_only":0,"data_only":0,"master_data":8,"client":0},"versions":["0.90.12"],"os":{"available_processors":32,"mem":{"total":"188.7gb","total_in_bytes":202616012800},"cpu":[{"vendor":"AMD","model":"Opteron","mhz":2599,"total_cores":4,"total_sockets":4,"cores_per_socket":1,"cache_size":"512b","cache_size_in_bytes":512,"count":8}]},"process":{"cpu":{"percent":115},"open_file_descriptors":{"min":1356,"max":1537,"avg":1429}},"jvm":{"max_uptime":"8d","max_uptime_in_millis":693602733,"versions":[{"version":"1.6.0_30","vm_name":"OpenJDK 64-Bit Server VM","vm_version":"23.25-b01","vm_vendor":"Sun Microsystems Inc.","count":8}],"mem":{"heap_used":"59.5gb","heap_used_in_bytes":63949850024,"heap_max":"127.7gb","heap_max_in_bytes":137160032256},"threads":621},"fs":{"total":"94.4gb","total_in_bytes":101461655552,"free":"48.9gb","free_in_bytes":52530081792,"available":"44.1gb","available_in_bytes":47376134144,"disk_reads":570224,"disk_writes":164405664,"disk_io_op":164975888,"disk_read_size":"16.1gb","disk_read_size_in_bytes":17347735552,"disk_write_size":"612.6gb","disk_write_size_in_bytes":657792794624,"disk_io_size":"628.7gb","disk_io_size_in_bytes":675140530176,"disk_queue":"0","disk_service_time":"0.9"},"plugins":[{"name":"bigdesk","description":"No description found for bigdesk.","url":"/_plugin/bigdesk/","jvm":false,"site":true},{"name":"hq","description":"No description found for hq.","url":"/_plugin/hq/","jvm":false,"site":true},{"name":"head","description":"No description found for head.","url":"/_plugin/head/","jvm":false,"site":true}]}}
    
    Quite human unreadable. Let's try now with jq:
    curl -XGET es1.grey:9200/_cluster/stats | jq "."
    Seems more elegant now:
    {
      "nodes": {
        "plugins": [
          {
            "site": true,
            "jvm": false,
            "url": "/_plugin/bigdesk/",
            "description": "No description found for bigdesk.",
            "name": "bigdesk"
          },
          {
            "site": true,
            "jvm": false,
            "url": "/_plugin/hq/",
            "description": "No description found for hq.",
            "name": "hq"
          },
          {
            "site": true,
            "jvm": false,
            "url": "/_plugin/head/",
            "description": "No description found for head.",
            "name": "head"
          }
        ],
        "fs": {
          "disk_service_time": "0.3",
          "disk_writes": 164465520,
          "disk_reads": 570266,
          "available_in_bytes": 47369240576,
          "available": "44.1gb",
          "free_in_bytes": 52523188224,
          "free": "48.9gb",
          "total_in_bytes": 101461655552,
          "total": "94.4gb",
          "disk_io_op": 165035786,
          "disk_read_size": "16.1gb",
          "disk_read_size_in_bytes": 17348456448,
          "disk_write_size": "612.8gb",
          "disk_write_size_in_bytes": 658024804352,
          "disk_io_size": "628.9gb",
          "disk_io_size_in_bytes": 675373260800,
          "disk_queue": "8.3E-5"
        },
        "jvm": {
          "threads": 621,
          "mem": {
            "heap_max_in_bytes": 137160032256,
            "heap_max": "127.7gb",
            "heap_used_in_bytes": 53713885664,
            "heap_used": "50gb"
          },
          "versions": [
            {
              "count": 8,
              "vm_vendor": "Sun Microsystems Inc.",
              "vm_version": "23.25-b01",
              "vm_name": "OpenJDK 64-Bit Server VM",
              "version": "1.6.0_30"
            }
          ],
          "max_uptime_in_millis": 694176092,
          "max_uptime": "8d"
        },
        "process": {
          "open_file_descriptors": {
            "avg": 1432,
            "max": 1546,
            "min": 1355
          },
          "cpu": {
            "percent": 134
          }
        },
        "os": {
          "cpu": [
            {
              "count": 8,
              "vendor": "AMD",
              "model": "Opteron",
              "mhz": 2599,
              "total_cores": 4,
              "total_sockets": 4,
              "cores_per_socket": 1,
              "cache_size": "512b",
              "cache_size_in_bytes": 512
            }
          ],
          "mem": {
            "total_in_bytes": 202616012800,
            "total": "188.7gb"
          },
          "available_processors": 32
        },
        "versions": [
          "0.90.12"
        ],
        "count": {
          "client": 0,
          "master_data": 8,
          "data_only": 0,
          "master_only": 0,
          "total": 8
        }
      },
      "indices": {
        "segments": {
          "memory_in_bytes": 3169089995,
          "memory": "2.9gb",
          "count": 2023
        },
        "count": 31,
        "shards": {
          "index": {
            "replication": {
              "avg": 1,
              "max": 1,
              "min": 1
            },
            "primaries": {
              "avg": 4.064516129032258,
              "max": 5,
              "min": 2
            },
            "shards": {
              "avg": 8.129032258064516,
              "max": 10,
              "min": 4
            }
          },
          "replication": 1,
          "primaries": 126,
          "total": 252
        },
        "docs": {
          "deleted": 188053,
          "count": 162713334
        },
        "store": {
          "throttle_time_in_millis": 172306131,
          "throttle_time": "1.9d",
          "size_in_bytes": 47552519067,
          "size": "44.2gb"
        },
        "fielddata": {
          "evictions": 0,
          "memory_size_in_bytes": 2858939869,
          "memory_size": "2.6gb"
        },
        "filter_cache": {
          "evictions": 504563,
          "memory_size_in_bytes": 5181806342,
          "memory_size": "4.8gb"
        },
        "id_cache": {
          "memory_size_in_bytes": 0,
          "memory_size": "0b"
        },
        "completion": {
          "size_in_bytes": 0,
          "size": "0b"
        }
      },
      "status": "green",
      "cluster_name": "gc-pb-grey",
      "timestamp": 1418914164326
    }
    
    
    Ok, this does not changes much however, it's easy to get json formatted. But try this: curl -XGET es1.grey:9200/_cluster/stats | jq ".nodes.jvm" and we get:
    {
      "threads": 619,
      "mem": {
        "heap_max_in_bytes": 137160032256,
        "heap_max": "127.7gb",
        "heap_used_in_bytes": 54125025648,
        "heap_used": "50.4gb"
      },
      "versions": [
        {
          "count": 8,
          "vm_vendor": "Sun Microsystems Inc.",
          "vm_version": "23.25-b01",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "version": "1.6.0_30"
        }
      ],
      "max_uptime_in_millis": 694408309,
      "max_uptime": "8d"
    }
    
    I took the outer object by first dot, 'nodes' property, and from inside it 'jvm' property. It is now trivial to get actual jvm version: curl -XGET es1.grey:9200/_cluster/stats | jq ".nodes.jvm.versions[0].version" Tadam:
    "1.6.0_30"

    Wednesday, January 22, 2014

    can your language talk json?

    Recently I discovered Python. It's really amazing that the language is that flexible so it can treat json literals as it's usual code. Think about constructing all objects in Java with any JSON library. Is this flexibility specific to dynamic languages? Btw I have found an example that well known and loved ;> C++ also is flexible enough to handle JSON:
    std::string json_cxx = { 
      key: "value", 
      number: 53     
    }; 
    
    This will compile. But... don't be fooled by compiler letting pass that code! The result is not what javascript developer could expect :/

    Thursday, June 13, 2013

    compiler could see that...

    I recently have met one of those not straightforward exceptions in tests. Something I first classified as 'perhaps it's config or classpath issue'. Here comes the stacktrace:
    java.lang.NullPointerException
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.junit.internal.runners.ClassRoadie.runAfters(ClassRoadie.java:72)
     at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:47)
     at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.run(PowerMockJUnit44RunnerDelegateImpl.java:133)
     at org.powermock.modules.junit4.common.internal.impl.JUnit4TestSuiteChunkerImpl.run(JUnit4TestSuiteChunkerImpl.java:112)
     at org.powermock.modules.junit4.common.internal.impl.AbstractCommonPowerMockRunner.run(AbstractCommonPowerMockRunner.java:57)
     at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
     at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
     at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
     at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
     at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
     at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
     at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
    
    When I looked at the tests I was able to see all test methods are successful, but after there is an error. I was able to figure out that NPE is thrown during invocation of method within clean up method that should be run after test. Here is the code:
     @AfterClass
     public void resetConf() throws Exception{
      setTestConfiguration();
     }
    
    Can you see that?
    In debugger I have found that first argument of invoke() is null.
    But it's quite ok. If it's null it's just static invocation and that's what I would expect during @AfterClass call...
    Yes. Fine.
    But resetConf() must be static then, you stupid reflection!