What? A blog?

Microservices in a Nutshell

posted Mar 25, 2016, 2:11 PM by Daniel Gomes   [ updated Mar 25, 2016, 2:20 PM ]

This is a great lecture about software architecture, design and Microservices:

https://www.thoughtworks.com/insights/blog/microservices-nutshell

(mind the video!)

And I'll just leave this here:


Useful foreword on design and specification documents

posted Dec 10, 2015, 12:17 AM by Daniel Gomes

"Shall" and "shall not" identify requirements to be followed strictly to conform to this document and from which no deviation is permitted.

"Should" and "should not" indicate that one of several possibilities is recommended as particularly suitable, without mentioning or excluding others, that a certain course of action is preferred but not necessarily required, or that (in the negative form) a certain possibility or course of action is discouraged but not prohibited.

"May" and "need not" indicate a course of action permissible within the limits of the document.

"Can" and "cannot" are used for statements of possibility and capability, whether material, physical or causal.

Creating a spanned GZ file

posted Jun 28, 2013, 5:53 PM by Daniel Gomes

I need to backup a large file from Linux to a VFAT file system, which has a file size limitation of 4 GB. So, I'm taking this note for future reference in case I need to restore the compressed file later.

Snatched from StackOverflow:

# create archives
$ gzip -c my_large_file | split -b 1024MiB - myfile_split.gz_
# uncompress
$ cat myfile_split.gz_* | gunzip -c > my_large_file

Quick *nix (mostly Linux) Health Checks #1

posted Apr 12, 2013, 8:24 AM by Daniel Gomes   [ updated Apr 12, 2013, 8:39 AM ]

Once upon a time, I spent a whole day trying to convince a junior system administrator to run lsof and tell me the size of largest file open because my app was performing really bad and the file system was filling up no matter what.

Since he was clueless about what lsof does and why a insanely large file ruins your performance, he just ignored me until things became so ugly, he got escalated and his boss provided me the lsof results (Yeah, I could not just become root and do it. Large corporations have these separation of duty rules).

I identified the evil process and killed it, recovering app's performance. Kudos!

So, kids, run lsof whenever your application goes weird. It may be looping through a file descriptor and writing huge amounts of data to the disk!

It must be run as root:

# 1 - Check top 30 processes with number of files opened:
sudo lsof | awk '$5 == "REG" {freq[$2]++ ; names[$2] = $1 ;} END {for (pid in freq) print freq[pid], names[pid], pid ; }' | sort -n -r -k 1,1 | head -30

Get used to your app's regular number of opened files and stay tuned for any unusual large number!

# 2 - Check the file sizes ordering by size, descending:

sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 30

Get used to your app's regular opened files size. Of course this varies a LOT from app to app, but usually after taking care of some app for sometime you get used to BAU parameters. Also, if you see a 5,000 Gb file and your app is not launching any nuclear missile, hey, c'mon...

(One liners stolen from http://thegoogleof.blogspot.com/2011/11/lsof-sort.html - I could do it if I want! Anytime! :P)


Now let's talk memory. The free command is your friend, a complicated friend but still a friend.

[root@xyz ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          7596       5475       2121          0         21        532
-/+ buffers/cache:       4921       2675
Swap:         8191       1138       7053

-m is to display data in megabytes.

So, I have 2,121 Mb free, right? Wrong. I have 2,675 Mb. And the OS is really using 4,921, not 5,475 (because this includes disk caching which is freed whenever your apps need RAM).

Of course this is a bit more complicated because it includes kernel slab reclaimable and 'free' command behavior, that ignores it. For a real good post on memory, read here:

http://stackoverflow.com/questions/3784974/want-to-know-whether-enough-memory-is-free-on-a-linux-machine-to-deploy-a-new-ap/4417121#4417121

If you need specifics on a process (Linux), get its PID, example:
[root@xyz ~]# ps aux | grep java | grep -v grep
joe  22209  3.3  1.8 954764 143336 ?       Sl   07:56   8:39 /opt/ibm/lotus/notes


Then cat its info from /proc:

[root@xyz~]# cat /proc/22209/status
Name:    notes2
State:    S (sleeping)
Tgid:    22209
Pid:    22209
PPid:    1
TracerPid:    0
Uid:    500    500    500    500
Gid:    500    500    500    500
Utrace:    0
FDSize:    1024
Groups:    18 498 500 502
VmPeak:     1028952 kB
VmSize:      954764 kB
VmLck:           0 kB
VmHWM:      228884 kB
VmRSS:      143336 kB
VmData:      601076 kB
VmStk:         100 kB
VmExe:          16 kB
VmLib:      176108 kB
VmPTE:         868 kB
VmSwap:       52096 kB
Threads:    103
SigQ:    1/60589
SigPnd:    0000000000000000
ShdPnd:    0000000000000000
SigBlk:    0000000000300000
SigIgn:    0000000000301000
SigCgt:    20000001c20864ff
CapInh:    0000000000000000
CapPrm:    0000000000000000
CapEff:    0000000000000000
CapBnd:    ffffffffffffffff
Cpus_allowed:    ff
Cpus_allowed_list:    0-7
Mems_allowed:    00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:    0
voluntary_ctxt_switches:    441126
nonvoluntary_ctxt_switches:    26452


Of course there's a lot of info there you don't need. VmPeak is a good one to know what's the maximum size your app ate from the system.

For detailed description on this:
http://www.lindevdoc.org/wiki//proc/pid/status

Perl - Thread it right!

posted Apr 4, 2013, 3:12 PM by Daniel Gomes   [ updated Apr 4, 2013, 3:25 PM ]

Ok, let's talk about Perl Threads.

I've seen so many dangerous scripts out there and sometimes I have to re-write them from scratch. People don't know SysV signaling and implements all sorts of crazy "technical gimmicks" causing potential (sometimes real) system instability.

If you're clueless about Perl Threads, well... start here: http://perldoc.perl.org/threads.html


Care for a "safe" Perl Threaded script? Check the skeleton below, I spent a couple of hours from my afternoon to produce a clean template. Just a few code comments so it doesn't get polluted:

-------->8-------- code snippet
-------->8--------

#!/usr/bin/perl -w

use strict;
use warnings;
use threads 'exit' => 'threads_only';

# This global variable is used only within threads
my $thread_id;


# This piece of code is executed only in the threaded process

# Signal handler for thread
sub thread_stop {
    print "Stopping thread " . $thread_id . "\n";
   
    $thread_id = "stop";
}

# Thread entry point - this is the worker sub
sub start_thread {

    # 1st thing we do is to bind the handler to the signal
    $SIG{'INT'}='thread_stop';
   
    $thread_id = $_[0];
   
    # Notice that each thread has its own $thread_id (refer to Perl
    # documentation about
interpreter threads
    # The thread scope is NOT shared by default
    while ($thread_id ne "stop") {
        # THREAD WORK! Done here!!
        print('Thread ' . $thread_id . ' - ' . localtime() . "\n");
        sleep(1);
    }
   
    return (0);
}

# From this point below...
# This piece of code is executed only in the parent process

# Signal handler for the parent process
sub main_handler {
   
    print "Finishing threads...\n";
    foreach my $thread (threads->list(threads::running)) {
        $thread->kill('INT');
    }
}

# Bind SysV signals for the parent - notice the handler (sub), different from the thread
$SIG{'TERM'}='main_handler';
$SIG{'INT'}='main_handler';

# Create threads and they'll start running afterwards
foreach my $tid ("one", "two", "three", "four") {
    threads->create('start_thread', $tid);
}

# At this point, threads are running in parallel

do {
    sleep (1);

    # Joining threads is good to have a clean flow
    # But joining is really useful only when the thread
    # is supposed to return a value
    my @joinable = threads->list(threads::joinable);
    foreach my $thread (@joinable) {
        print "Joining threads...\n";
        $thread->join();
    }
} while (threads->list());

print "Exiting...\n";

exit (0);


-------->8-------- code snippet -------->8--------

Execute the script above. It will start to print timestamps. To make it to stop just hit CTRL-C.

Sample output:

$ ./thread_test.pl
Thread one - Thu Apr  4 17:06:02 2013
Thread two - Thu Apr  4 17:06:02 2013
Thread three - Thu Apr  4 17:06:02 2013
Thread four - Thu Apr  4 17:06:02 2013
Thread one - Thu Apr  4 17:06:03 2013
Thread two - Thu Apr  4 17:06:03 2013
Thread three - Thu Apr  4 17:06:03 2013
Thread four - Thu Apr  4 17:06:03 2013
Thread one - Thu Apr  4 17:06:04 2013
Thread two - Thu Apr  4 17:06:04 2013
Thread three - Thu Apr  4 17:06:04 2013
Thread four - Thu Apr  4 17:06:04 2013
^CFinishing threads...
Stopping thread one
Stopping thread two
Stopping thread three
Stopping thread four
Joining threads...
Joining threads...
Joining threads...
Joining threads...
Exiting...
$


Creating an Entity Relationship Diagram from Existing Databases

posted Mar 14, 2013, 11:18 AM by Daniel Gomes   [ updated Mar 14, 2013, 11:23 AM ]

At the graduation we learn that every system must have proper documentation and this documentation must be provided to the product support teams after the project is closed, so people can properly support the application.

Bad news. Even at big companies, big technology companies, documentation is still very poor.

So, to easy your pain, whenever you need to generate reports or understand the data layer, Oracle has made freely available a great feature in SQL Developer to reverse engineer an Oracle database and output modeling information.

First, download Oracle SQL Developer:

http://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html

Follow installation instructions over there. Then, follow these steps I've got from here: http://stackoverflow.com/questions/6580529/how-to-generate-e-r-diagram-using-oracle-sql-developer:

File → Data Modeller → Import → Data Dictionary → select DB connection (add if none) → Next → last few steps intuitive.

Remark: It's very important to select the correct schema.

It will take some time analyzing your database. If the database lacks proper constraints, PKs and FKs it may not be able to fully represent the relationships. Also, the layout it outputs from the data dictionary is somewhat messy and you may want to organize it according to the affinities between tables.

Note #1: It can also be done with Oracle SQL Developer Data Modeler standalone.
Note #2: You can also import several other formats like DDL files. I'm not sure if it would be useful with MySQL data dictionaries, for example, but might worth a try.
Note #3: in the "select DB connection" step you'll realize it doesn't share connections with SQL Developer main module. You have the option to 'import' them, though. Go figure...

Common SQL JOIN Patterns

posted Oct 5, 2012, 12:48 PM by Daniel Gomes   [ updated Oct 5, 2012, 12:55 PM ]


This chart is something I really need from time to time so I'll just leave it here for further reference. :)

Lots of people go crazy on SQL statements sometimes just to reproduce one of these patterns. They even dare to write fancy PL/SQL scripts because they don't fully understand how to efficiently fetch the information by applying one of these patterns.

Oracle spent zillions of dollars to optimize the engine so you don't have to manually write procedures or complex queries for these cases! Use the respective join clause! Sometimes you even code worst logic with higher O(n) than if you let Oracle (or your preferred engine) to optimize it.

For example, consider you want all records from table A that has no related record in table B (the second case, top to bottom at the left side). You can do it both ways efficiently:

SELECT * from tran a LEFT JOIN tran_acty b ON a.id = b.id WHERE b.id is NULL;

SELECT * FROM tran a WHERE NOT EXISTS (SELECT 0 FROM tran_acty b WHERE a.id = b.id);

Well, as you can see, using the LEFT JOIN is more elegant since it has no need for a sub select and code is cleaner.

Regarding speed and optimization, both cases have the same execution plan so efficiency wise it doesn't matter.

But I've seen people writing Perl scripts just to do the same... well, see my point? :)

Leaders are made, not born

posted Oct 3, 2012, 10:29 AM by Daniel Gomes   [ updated Oct 3, 2012, 10:36 AM ]

I snatched the following article from Daniel Goleman, I hope he doesn't mind since it was public published in Linkedin.

It caught my attention since one of my worst professional experience ever was upon a time when I tried to coach a person to experience the spotlights a bit because I saw great leadership potential in that person.

The conversation derailed and turned out into an awkward situation. Of course I blame myself since I totally failed in coaching skills that time. So, I could have used something like this.

----
Leaders are made, not born
October 02, 2012 - By Daniel Goleman, Board Member at Mind & Life Institute
Original available at http://www.linkedin.com/today/post/article/20121002124348-117825785-leaders-are-made-not-born

I know someone who has terrific ideas, a natural innovator and smart strategic thinker. But she never speaks up in meetings. She's too shy, she tells me. And, she adds, she's been like that all her life: "That's just who I am."

So she'll quietly share her thoughts with the person next to her, who quite often will voice them to everyone - and get the credit for her sound insights. It drives my friend crazy.

On her own, she's a dynamite performer. And that has caught the eye of management. She's been told she may get a promotion to team leader. She like's the idea, except for one thing: she's petrified at the thought of having to speak in front of everyone.

Is her potential career as a leader doomed? Not at all. Leaders are made, not born. Let's take it from the start, childhood. Parents will type their kids from day one as "bold" or "timid". And, they assume, once shy, always shy.

But now careful research has debunked those assumptions. A study at the University of Wisconsin that followed 70 toddlers for several years tested them on how bold or shy they were at three ages: three, seven and nine. The result: kids who were shy at three were not especially likely to be shy at nine. Kids went back and forth, from bolder to more timid, or the reverse, at what amounted to random.

What made the difference was not some set of genes that permanently fixed them as outgoing or shy, but rather what they experienced in their lives. If parents, say, regularly exposed a shy kid to strange playmates at a playground and was emotionally supportive in the face of their uncertainty, those kids learned a crucial lesson: I may feel a little afraid at first, but if I go ahead and engage despite the fear, I'll have a great time.

And the same holds for grownups. Someone a bit timid in groups at work can overcome her shyness with sustained effort. The steps to developing key leadership abilities like self-confidence are well-known.

First, ask yourself: do I care? You need to be motivated, because it takes effort and time - and may be a bit uncomfortable at first.

Second, ask, do I really need to improve? Get opinions of people you respect and trust about what they see as your strengths and limits when it comes to leadership potential.

Third, answer the question: what should I practice? It helps to have a learning plan, a contract with yourself. In this case it might be: Whenever the opportunity arises, I will speak up and say what I'm thinking, rather than just sitting there being quiet.

Fourth, find a friend to support you, someone you can talk over times you don't get it right. Think through what kept you quiet, and what you might do next time to have the confidence to speak up.

Finally, seize every opportunity to practice - not just work, but anywhere in your life it may happen. If you follow this regime, you'll be reprogramming your brain in a way that will make it easier and easier to overcome that still, small voice that says, "Be quiet."

My quiet friend has been trying it. And, she tells me, she's starting to like taking the lead.

----

1-8 of 8