Raise the fooBar Rotating Header Image

Eclipse & Git: Mind your Windows line endings!

Recently, our team at work stumbled upon a strange behavior in their Git projects: there were files with Windows CRLF line endings checked into the git repository. If you know git, you’re aware of the line ending settings and that line endings basically should never be an issue. Well, turns out: it is, and it is quite often.

How it should work

While investigating the issue, I found the excellent Mind the End of Your Line article over at adaptivepatchwork.com, which explains how line endings should be handled in the first place. It also helped me find the cause of the issue, so props to them!

The most important bit in this case is the core.autocrlf setting. As every git setting, it can be specified per-repository or globally (per OS user). On Windows, it seems to be recommended setting it to true, causing git to use Windows CRLF in the workspace and auto-converting line endings to LF when committing to git’s object database. Going with the input setting also shouldn’t cause too much trouble: in this case, git checks line endings out into the workspace as they are in the object database without changing them, but still makes sure to replace any Windows CRLF line endings with linux LF when committing to git’s object database.

The only core.autocrlf setting to avoid at all cost, at least on windows, is false. This would cause git to commit line endings into its object database exactly as they appear in the workspace (the only setting letting Windows’ CRLF into git’s object database).

Our setup

We are mostly dealing with java projects, but our repositories contain a bunch of shell scripts for automation, testing and stuff like that. Our development machines are Windows workstations while test and production servers are linux/unix. We use Eclipse along with its egit plugin for development.

The symptom

I once had to merge a huge feature branch. After resolving merge conflicts and committing my changes, I noticed that pushing the merge took longer than usual. Also, the git repository had nearly doubled in size. It turned out that my merge just changed every single text file’s line endings to Windows CRLF in the repository.

I fixed it by resetting everything to linux LF line endings. However, after a couple of months, we noticed that Windows line endings were slowly crawling back into the repository – apparently cumulating through multiple developers’ commits and merges over time.

Pinpointing the problem

This turned out to be a nightmare. It took an awful lot of time to reproduce the problem, let alone find a logical explanation to why this kept happening.

Whenever I ran git config core.autocrlf in a repository, it always yielded the expected input, so I didn’t suspect the autocrlf option and didn’t care that git config --global core.autocrlf wasn’t set. I ended up creating a test git repo and running a bunch of merges with intentional conflicts before I was able to pinpoint the exact constellation causing the problem:

As it turns out, it is a combination of the git spec, the windows git command-line and Eclipse’s egit plugin causing the issue.

  • The git documentation states that settings are either found in the current repository’s ./.git/config file or the global %HOME%\.gitconfig, the latter being the fallback if they’re not found in the former. It also specifies that the default value for the core.autocrlf setting is false – remember? the worst setting there could be on Windows if line endings matter…
  • The Windows git client (from git-scm.com) does ask for the core.autocrlf setting during setup to make sure you have it set, but it doesn’t seem to store it in the documented global location (%HOME%\.gitconfig). Instead, it seems to store it somewhere else (I haven’t found where though) and uses this location as ultimate fallback during command-line git invocations when the setting isn’t found in the usual locations. As long as you only ever use the command-line git, you’re fine, however…
  • Eclipse’s egit plugin doesn’t use the command-line git client directly but comes with its own implementation, which does honor the documented locations and settings. However, as the “magical ultimate fallback” used by the Windows git command-line client seems to be undocumented, Eclipse couldn’t possibly know about it. So from Eclipse’s point of view: No core.autcrlf in the repository, no core.autocrlf in %HOME%\.gitconfig: use the default value: false. If you then happen to commit merges, or even worse, if you previously cloned a repository with core.autocrlf set to true, your Eclipse will end up committing everything with Windows line endings while your command-line git behaves perfectly fine.

I checked with some colleagues from other teams and they also had the global core.autocrlf setting unset – and incidentally, their Eclipse was behaving strangely in the git synchronize view. After they set the global setting, everything was fine. So it looks like manual action is needed to make it work correctly.

Oh, and to make things even worse: I currently suspect something to overwrite my %HOME%\.gitconfig, removing the core.autocrlf setting as it magically disappeared somewhere in the last weeks. I’m not quite sure what it is tough…

In conclusion, TL;DR

If you use Eclipse and Git on Windows, make sure you have the core.autocrlf option globally set to true AT ALL TIMES!

To check for the option:

    git config --global core.autocrlf

(this should yield true).

To set it:

    git config --global core.autocrlf true

May this post help other poor souls like me! 😉

– Cheers!

Die Technik bei Pro7’s “Keep Your Light Shining” – ein Kommentar

Gleich 2 deutsche TV-Sender haben in diesem Monat neue TV-Formate an den Start gebracht. Was so besonders daran ist? Sie binden die TV-Zuschauer überaus interaktiv und umfangreich über das Internet in’s Geschehen der Sendung ein.

Zum einen war da das Quizduell. Die ARD hatte den Vortritt… und hat gleich mal in’s Klo gegriffen. “Ein Hackerangriff”, so hieß es, habe die Server lahmgelegt. Im Nachhinein ist allerdings gar nicht so klar, ob es sich nicht doch nur um die zu erwartende hohe Last einer TV-Livesendung gehandelt hat. Besonders zu betonen ist allerdings, dass alle Beteiligten den Fail sportlich und mit Humor genommen haben. So kann ich jemandem einen solchen Fehltritt auch gerne verzeihen, ist ja schließlich alles #Neuland 😉 Etwa eine Woche später war dann technisch alles behoben und die Show konnte wie geplant mit Beteiligung der Internet-Nutzer stattfinden.

Heute waren dann die privaten Sender an der Reihe. Vertreten durch Pro7 mit der Musik-Show “Keep Your Light Shining”. Aus Spaß und Neugier habe ich heute tatsächlich mal eine app an mein Facebook-Profil gelassen und per Browser mitgemacht.

Ich war von der Technik von Anfang an begeistert. Wie die Entwickler es hinbekommen haben, die Web-Applikation so synchron zum TV-Bild zu halten ist wirklich bemerkenswert. Die Show funktioniert so: Es wird in jeder Runde ein Lied gesungen. Von allen Kandidaten abwechselnd, im 30-Sekunden Takt. Die Webseite aktualisierte bei jedem Kandiaten-Wechsel die Anzeige entsprechend… und das, zumindest für mich, fast auf die Sekunde genau synchron zum TV-Bild. Komplett in HTML5, ganz ohne Audio-Watermarking und sogar ohne WebSockets.

Da ich in dem Bereich ja kein ganz unbeschriebenes Blatt bin, habe ich mir während der Sendung die von Pro7 eingesetzte Technik mal etwas genauer angeschaut und ein paar best practices abgeleitet 🙂

Als View-Layer wurde Facebook’s React eingesetzt. Damit wurde die Mechanik der Applikation größtenteils im Voraus in JavaScript programmiert. Eine JSON Konfigurations-Datei hat der Applikation die Kandidaten der Woche mit allen Details wie Name, Bilder usw. mitgeteilt (so muss nur diese eine Datei für die nächste Episode der Sendung ausgetauscht werden). Es sind also in erster Linie eine handvoll statischer Resourcen involviert gewesen: die kleine HTML-Datei, React selbst, die KYLS-JavaScript-Applikation, die JSON Kandidaten-Konfiguration, ein bisschen CSS und ein paar Bilder wurden via Amazon’s CloudFront verteilt. Durch dieses hochperformante CDN ist das Ausliefern der statischen Resourcen kein Problem.

Bleibt noch die Synchronisation und das tatsächliche Voting.

Die Synchronisation wurde, soweit ich das beurteilen konnte, durch “stinknormales” 10-Sekündiges Polling realisiert. Zu erwähnen ist hier, dass beim Polling auf jedes Byte geachtet wurde: Übertragen wurde ein JSON-Objekt mit ein-buchstabigen keys und nur numerischen IDs und arrays als Werten. Das JSON Polling übermittelt den Stand der Sendung (aktuelle Runde, wer in dieser Runde noch drin ist, wer gerade singt,…) an die React Applikation. Was hier auf Serverseite verwendet wurde, kann ich natürlich nicht wissen. Ich weiss nur, dass sich als Webserver ein “Apache” gemeldet hat, und dass auch bei dieser Applikation auf Amazon’s Cloud gesetzt wurde. Per DNS RoundRobin kamen gut drei Dutzend Amazon-Server rein (danach habe ich mit Anfragen aufgehört ;)) – die Cloud halt. Mich würde aber wirklich interessieren, was hier serverseitig für die Persistenz bzw. als Datenstore genommen wurde…

Das Voting, also Daumen-Hoch oder Daumen-Runter Clicks, wurde, ähnlich wie das Polling, mit einem eigenständigen XMLHttpRequest an die Amazon-Server realisiert. Dieser beinhaltete die Runde und die Kandidaten-ID als URL Parameter. Ich vermute, die eindeutige User-ID (vom Facebook-Konto) war in irgendeiner Form in den Cookies enthalten. Keine Magie, alles straight forward.

Fazit: so viel wie möglich statisch machen, möglichst kein dynamisches HTML auf Serverseite rendern sondern JS-Client-Libraries wie React verwenden, den statischen Content cachen und auf CDN’s setzen, skalierbare Applikationsserver (zB Amazon’s Cloud) für Interaktionen verwenden, bei Live-Updating/Polling jedes Byte sparen.

Mir hat die Sendung jedenfalls aus technischer Sicht, Spaß gemacht! Ich bin gespannt was für Internet-Formate sich die Sender weiterhin so ausdenken, was davon technisch hinhaut, und wie es gebaut sein wird 🙂

How to delete entire twitter conversations (direct messages)?

I recently wanted to delete a massive twitter conversation with thousands of messages. To my surprise, there’s no way to delete a whole conversation with a single click in the Twitter.com Web GUI. The Twitter MacOS App has a “Delete Conversation” entry in a conversation’s context menu, but that just removes it from the Gui. Restarting the app shows that it is, in fact, still present.

My guess is that their Message storage doesn’t allow efficient deletion of conversations. But what if I _really_ want to get rid of them?

An option is to search for these services… you know… those shady ones that claim they can do the job for you, want access to your twitter account and, after you have granted them full access, ask you to take out your wallet and pay if you _really_ want them to help you. Well, F such a business model. Seriously.

There’s an easier, much safer way to delete entire conversations and guess what: I’ll tell it to you for free!

Okay, here’s what you do:

  1. Open Chrome (we’ll need the developer console. Other browsers might work, too, but I use Chrome, so I’m gonna stick with it here)
  2. Log in to twitter and go to your “Me” page
    https://twitter.com/<yournick>
  3. Open your messages (the button with the envelope) and select the conversation you want to delete. You now have the conversation-to-be-deleted in front of you.
  4. Open the JavaScript Developer console (Command+Alt+J on Chrome / MacOS)
  5. Paste the following JavaScript into the console and press enter:
    $('#global-actions').append($('<li id="del-script">').append($('<a class="js-nav">"').text('...')));
    window.delscrpt = setInterval(function() {
      $('.js-dm-error:visible').each(function() { $('a.js-dismiss', this).click(); });
      var count = $('.js-dm-delete').length;
      if (count < 3) {
        $('#del-script a').text('Del Done.');
        clearInterval(window.delscrpt);
        return;
      }
      var randIdx = Math.floor((Math.random()*count)+1);
      $('#del-script a').text('Del ' + randIdx + '/' + count);
      $('.js-dm-delete').eq(randIdx).click();
      $('.dm-delete-confirm .js-prompt-ok').each(function() { $(this).click(); });
    }, 2000);
  6. Sit back, relax and watch your conversation disappear one message at a time.

DISCLAIMER:

  • Don’t expect any support. Use at your own risk. I think this should be pretty obvious 😉
  • This method works by directly manipulating the HTML DOM of the twitter page (remote controlling the GUI if you will). It works for now (mid-September 2013), but if twitter changes their homepage, this method will die. Keep that in mind.
  • The deletion might take some time. We don’t want to hammer the twitter servers and get our account blocked after all 😉
  • Twitter has some major issues with message deletion (internal server error’s and stuff). That’s why we can’t delete messages in order and have to delete them randomly until the conversation is empty. You can’t just delete parts of a conversation. You’ve been warned.
  • The script will stop when less than 3 messages are left in the conversation, in order not to bug out. You’ll have to spend the 10 seconds and delete these last messages manually.
  • The script displays some numbers in the twitter navigation bar (in the background). The first number is the index of the randomly selected message that’s being deleted from within the returned message list. The second is the total number of messages currently returned by the twitter servers. This total number doesn’t represent the “real” number of left-over messages all the time. That’s just because of the way twitter works and spits out your messages… Non-deterministic NoSQL and stuff…

Hope it’s useful to anyone 🙂

Eclipse: “Insufficient access privileges to apply this update”

If you’re trying to update your Eclipse on your magnificent Windows box* through the integrated update mechanism, and are seeing “Insufficient access privileges to apply this update” when you try to select an element, just run Eclipse with admin privileges and it should go through.

To quickly run Eclipse with admin privileges: open the start menu, search for “Eclipse”, point to it and run it with CTRL+SHIFT+Enter.

* WARNING: Irony