Getting better over time : Troubleshooting Chrome OS updates

One of the salient features of Chrome OS is its ability to do transparent updates with little or no interaction from the user. This not only ensures the user is always protected, it also improves performance and features over time.

https://youtube.googleapis.com/v/QtavfiBOv_g&source=uds

If you administer a fleet of chrome devices, I recommend you read this support on how to correctly configure this. It goes through all the autoupdate (AU) terminologies and suggests best practices which will help you in long run.

How to check if AU is properly working in your network

  1. I’ll strongly recommend to enable reporting using your admin console. This will allow devices to send its OS version to the reporting engine.
  2. Then use the Admin SDK APIs to generate reports for devices in your domain, grouped by major version. I recommend you write your own scripts, but you can try third party tools like these to understand what the APIs are capable of.
  3. If you have lots of different networks, try to generate separate reports for devices in different networks to see if any of them are further behind than others. 
  4. If you do notice some devices are not getting updates, look for the following failure patterns
    1. Check if the devices are being blocked from reaching update engine. White-list these domains if you need to implement ACLs.
    2. If you have enabled “scattered” updates, disable it for those users. “Scattered” updates is not good for devices which are not used every single day.
    3. If your devices never get updates, you may have to reach out to Google’s enterprise support team for deeper analysis.

    How to check what my devices should be at ?

    Google publishes all the version numbers for all of its chrome devices at this location:  http://cros-omahaproxy.appspot.com/. If you know what hardware you have, you should be able to find the latest stable, beta and dev version numbers there. Here is what it looks like for CR-48s and original Samsung chromebooks.

    Understanding AU Logs

    1. You can extract the AU logs from the device very easily using the following steps
      1. Go to chrome://net-internals#chromeos
      2. Click on “store debug logs”. This creates a compressed archive in the “Download folder”
    2. After exploding the compressed archive, look for files under “/var/log/update_engine” which has all the AU related logs
    3. Find the latest log file there and open it up in an editor
    4. The AU request would look something like this
         
         
             
             
         
    5. Interpretation of the request
      • Current version of the OS: 4319.96.0_i686
      • Hardware is : PARROT (find more info here)
      • Lang=”en-US” means its a US version of hardware
      • track=”stable-channel” means this is requested a stable update 
      • Lang=”en-US” means its a US version of hardware
      • track=”stable-channel” means its requesting a stable update 
    6. The AU response would look something like this
      [1121/083519:INFO:omaha_request_action.cc(717)] Omaha request response:
    7. Interpretation of response
      – status=”noupdate” means there are no updates available for this particular device
    8. If your device is getting “noupdate” even though its supposed to be on a newer version, something may be wrong. If this goes on for a few days, contact support to troubleshoot further.

    Putting Chrome OS behind SSL based webfilters

    Educational institutions, particularly the K12 are stuck between a rock and a hard place. They are always in search of ways to open up newer technologies to students, but don’t want to give up their ability to manage and filter what students can see or do. As a father of two I approve that.

    While Chrome OS does take security very seriously and tries very hard to discourage “man in the middle attack”, it does provide an industry tested feature to allow educators to filter web content for students in its recent version of Chrome OS. To understand how it works in Chrome OS, I’ll first explain how the Chrome OS works internally.

    Chrome OS devices, as most of you already know, has two distinct components. The Chrome browser is what provides most of the UI, but deep inside it also has an operating system built on top of linux. Among other things that OS is responsible for, auto-updates and security are two of the most important. 

    The web filtering feature which Chrome OS provides for our enterprise and schools users allows all “user session” traffic from the browser to be intercepted, but doesn’t allow any of the system requests to be intercepted in the same way.

    Network setup

    To get a chromebook to work correctly in an environment with webfilter, its important to let webfilter know which hosts chromebook would connect to for which it won’t tolerate SSL inspection. Google has published a set of domain names here which can be used for this purpose.

    Note that whitelisting by IP addresses (netblocks) is not good enough. The IP addresses mapped to these hosts keep changing and the only reliable way to whitelist them is by whitelisting the domain names as it is. Most webfilters (including some transparent webfilters) support this and if you are not sure, contact your proxy/webfilter provider to understand how to do it.

    Quick test

    Once the network is setup, import your custom root CA cert into the browser using certificate

    manager under “Authorities” and make sure you enable “Trust this certificate for identifying websites.” Then go to any website which you think should be intercepted and try to see if browser threw any error. Even if it didn’t throw an error, check at the certificate details and confirm that it was signed by your webfilter.

    Broader test

    Once the tests confirm that everything is working as expected, its time to do a broader test using management console. To prepare for this test, I would recommend picking a small set of users who are are ok with brief interruption (in case something goes wrong) and are willing to provide you with detailed feedback to help you debug the issue.

    In your admin panel, go to Chrome’s “Advance settings” section and then “Networks”. Pick the OU where all of your test users are and then click on “Manage Certificates” button on the

    top right corner. 


    Upload your certificate and check the box for “Use the certificate as an HTTPS certificate authority”. 

    The example on the right shows my setup where I’m using zscaler’s cloud based webfilter.

    The final test to make sure this is good, would be to move these users to a network where there is no direct network access.  Have them be forced to go through the proxy/webfilter and see if anything breaks. 

    Let this configuration stay like this for a few days/weeks and collect feedback on whether users noticed any other side effects. For example make sure devices are getting updates (which is critical) and that user users can be added any time.

    Complete the transition

    Once everything has been tested, apply the certificate to more and more OUs until the transition is complete.

    Caveats

    There are few caveats you should watch out for
    1. Even though this policy is being applied as a user policy, it will only work on devices which are enrolled to the same domain. This is one of the most common reasons for the feature not working.  This also means that if the device was unenrolled, it may cause network connectivity failures.
    2. Since this is a user policy, other users using the same device will not get this feature automatically. Each user has to be moved into an OU where this certificate is installed.

    More info

    Chrome device fleet reporting using APIs

    Chrome Devices have been a huge success in places like schools where students and teachers want the mobility and price point of a tablet but usability of a laptop. Like everything else Google focuses on scale and part of the Chrome Device offering to enterprises and schools are the tools around the devices to manage the fleet.
    Chrome Device management can be centralized which allows admins not only manage apps on the device, they can also push complex network and system settings with a touch of a few key strokes. And that works the same way for a customer with 10 devices and one with 20000.

    Recently Google released a new batch of APIs called the Admin SDK which includes some new APIs to discover and manage devices in the fleet. Using these APIs, admins can not only get a list of the active devices they have, they can also find out if the devices are having update issues. As an example of how to use this API, I’ve published a sample script called “ChromeOSDeviceStats” which I wrote for my own domain. This could be used to quickly get high level stats of devices in the domain, when they were enrolled, what version they are on, and what Orgs are they part off.

    The script could be further adapted to automatically send email reports like these when certain conditions are met. For example, if a school is interested in knowing about all recent “enrollments”, they could enhance this script to send that report.

    —————————————————————————
    Total devices : 31
    Total active : 31
    Total in Dev mode : 1
    Total devices enrolled in last 2 days : 0
    Total devices enrolled in last 7 days : 1
    Total devices enrolled in last 1 month : 2
    Total devices enrolled in last 1 year : 6
    Distribution of devices across models: 
                              Alex : 5 
                              Google_Stumpy : 2 
                              Google_Link : 6 
                              Google_Snow : 1 
                              Google_Lumpy : 8 
    Distribution of devices across versions: 
                              24 : 1 
                              25 : 5 
                              26 : 2 
                              27 : 5 
                              21 : 6 
                              23 : 3 
    Distribution of devices across orgUnits: 
                              /Eng : 3 
                              /hack : 1 
                              / : 31 
    —————————————————————————
     

    Openvpn in EC2 for Chromebooks : Part 2

    [ Update: take a look at this write up as well ]

    ChromeOS has a minimalistic design does a fairly good job at hiding the complex internals of the operating system. But deep inside it still runs linux and has a full blown openvpn client. In this post I’ll show you how to use “ONC” (Open Network Configuration) format to configure OpenVPN client inside ChromeOS. This file is very similar to an .ovpn file.

    The core scripts which did the cert conversion and created the sample ONC file was contributed by Ralph by reading the ONC documentations by himself. If there is a bleeding-edge user of chromebook.. he is the best example I can think off.

    These scripts are now published on github and here is a step by step guide of how to use it.

    Step 1: Launch a new amazon instance ( Based on Amazon Linux AMI )

    • Pick defaults for everything ( Use t1.micro for the cheapest instance )
    • If you don’t already have a keypair, create one yourself and upload your public key to amazon

    Step 2: Update security group you used. Allow UDP:1194 incoming.

    • Open 1194 UDP incoming 
    • Open 22 TCP incoming

    Step 3: Find the IP address of the new instance.

    • Find the “Public DNS” address. This is the address we will SSH to.

    Step 4: SSH into the server

    • If you only have a chromebook, use this extension to initiate ssh
    • Upload both private and public key to this extension.
    • Fill in the hostname and username (ec2-user)

    Step 5: Run the quick setup script

    curl https://raw.github.com/royans/ec2_chromeos_openvpn/master/quicksetup.sh > quicksetup.sh; 
    chmod +x quicksetup.sh;
    sudo ./quicksetup.sh email_address@blogofy.com
    • These three lines downloads the script and launches the setup script.
    • Remember to put in your email address instead of the one listed here
    • This step may take two to three minutes before it prompts you for anything

    Step 6: Select default values

    • When it does prompt you, just choose the defaults
    • If it asks you y/n questions… just select ‘y’ 
    • It will ask you for password a few times… just press enter (which sets no password on the keys)
    • Remember this is proof of concept and that you should customize it before you can use it for real stuff

    Step 7: Wait for the email. Download the ONC file attached.

    Step 8: Upload ONC file into chromebook

    • Make sure it says “ONC file successfully parsed” after the import.
    • [Advanced users: /var/logs/ui/ui.LATEST will have parse errors if you want to investigate a failed import]

    Step 9: Try to connect to the openvpn server

    • Just click on “Connect”

    Step 10: Connected

    • At this point you should see a solid (should not be blinking) VPN signal on the wifi icon. 

    Step 11: Verify if its traffic is being routed through amazon

    Patches: Pull or Push ?


    Most people prefer to disagree with the masses on whether they like sunny side up or scrambled eggs. And the form of getting patches is no different. If you ask an IT administrator (which is the person applying patches in most corporate organizations) they will tell you horror stories of how patches can go wrong and would be happy to give you examples of why every patch needs to be individually tested before deploying. 
    But my dad, for example, doesn’t care about patches, and while he won’t go out of the way to install a patch, he may be ok with patches being pushed to him automatically.  

    This debate  reminds me of another interesting debate in the Web-Operations world about “continuous deployment“. In that case the debate was whether applications should be deployed in scheduled releases (for example every quarter) or whether it should be released as things gets developed and pushed.

    If you think about this a little more it would be very clear that the developers who build the patches are the ones who first need to be ready to do “continuous deployment”. The confidence of every patch is directly proportional to the robustness of test infrastructure… and the unit-tests and integration-tests which are associated with it. For products where there are test-gaps it makes sense for IT administrators to constantly monitor and test every single patch before its deployed. 

    So that brings us back to “Pull or Push“. Because of the increase in number of recent attacks, I am now very conscious about what products I use on a daily basis and try very hard to pick those which can auto-update without nagging me. And they are usually the ones which have robust test infrastructure to allow “continuous deployment”… which in turn means that they are usually the ones which have better test coverage of their products and are the ones which can quickly patch something bad very quickly with confidence (like a 0-day).

    I do understand that mistakes in ‘Push’ based patches can be very expensive, but they are still more secure for end users when it comes to privacy and security.

    The reason why I was thinking about this is that Adobe released a patch today.. hope you noticed. I just wish it would auto-update devices where have it installed… its easy to forget to update devices, and in general not doing silent auto-update makes me worried that they are not super confident about their test infrastructure.

    Capturing wifi traffic of one station from another

    This is more of an embarrassing tale than a real how-to document. But I found this interesting enough that I don’t mind sharing it.

    A couple of weeks ago I was tasked to capture wifi traffic from a device which didn’t have any capture software built in and I wondered how one would do it.

    I have used sniffing tools on my Mac to passively sniff activity on access points around me. Because I’ve always tested such tools in places with dozens of access points with multiple saturated channels, I always assumed that all wifi stations ( laptops ) frequently switch channels. I also assumed that AP (Access points) which are setup to select channels automatically are designed to automatically switch channels anytime if they find a better (less noisy) frequency to provide services at.

    And because of those incorrect assumption, I concluded that sniffing another wifi station would be a difficult task because it would be impossible to dynamically change the channel of a second wifi station to follow the first one to correctly sniff all the packets.

    After a short discussion with a colleague I found out that most wifi stations don’t really switch AP points unless the noise to signal ratio gets too bad, and most APs never change channels once they are fully initialized.

    At the end, to sniff one device, all I had to do is keep the second device close to the first one and make sure that the second one joins the same channel as the first one. For my tests I used open wifi AP which were easier to capture/decode. At this point, if your hardware is capable of promiscuous mode and you have the right software for capture, u should be able to put in a filter with the mac address of the device you want to capture to initiate the process.

    Chrome Frame – How to add command line parameters

    Chrome frame intentionally does stuff without getting in the way of the user. This sometimes makes things harder to debug. For example how can one debug an issue if chrome frame doesn’t even launch ? Apparently there is a flag for that. But you have to know how to enable it. Here are the steps.

    1. Make sure chrome frame is installed.
    2. We can enable startup flags for dumping debug logs using a policy called AdditionalLaunchParameters
    3. If this is just for one desktop, I recommend doing a registry edit (it can be pushed via GPO as well)
    4. Add a REG_SZ property “AdditionalLaunchParameters” to “SoftwarePoliciesGoogleChromeAdditionalLaunchParameters” with the value “–enable-logging –v=1” (also documented here and mentioned here)  [ Attachment 1 ]
    5. Next kill the IE browser and make sure chrome is also dead by checking taskmgr
    6. Restart IE and go to “gcf:about:version” and confirm that the parameters you added show up next to “Command Line:”. If this doesn’t work… skip this step and go to next step anyway.
    7. Under your “Application Data” folder (its inside “Documents and Settings”) search for a file chrome_debug.log [ Its usually in “Application DataGoogleChrome FrameUser DataIEXPLORE

    Chromebooks with Openvpn on EC2

    Chromebooks are perfect companions for travel. They are light, secure and one generally doesn’t have to worry about data theft in case they loose the device.  But surfing from hotels and coffee shops is another story. While most sites are in SSL, there are enough websites which are not… and even the ones which support SSL sometimes forget to use SSL connectivity for sensitive data. Which is why extensions like “HTTPS everywhere” is highly recommended.


    If I could, I’d pay a few cents for extra level of privacy when using these public wifi networks. In this post I’ll document how you could quickly setup an openvpn server on EC2 instance to do exactly this for your chromebook.

    Prerequisites 

    1. A working EC2 account
    2. A working key-pair (required to ssh into the EC2 instance)
    3. Chromebook with R23 or later 

    Step 1 – Launch Amazon Linux AMI ( I used 32 bit for my setup.. its the cheapest). Pick all the defaults options and pay attention to which “Security Group” you would be selecting. It would most probably be called “default”

    Step 2 – Edit the security group used by the instance and make sure 1194 udp is added to “inbound” port list.

    Step 3 – Ssh into the EC2 instance using your key ( you could also use this extension if you have the ‘identity’ file instead of the .pem)

    ssh -i my_key.pem ec2-user@ec2-75-101-188-186.compute-1.amazonaws.com

    Step 4 – Add a user, set password and update the server

    sudo bash 

    useradd temp 

    echo ‘my_password‘ | passwd temp –stdin 

    yum -y update

    Step 5 – Install/start openvpn server with basic options

    # Install  

    yum -y install openvpn

    yum -y install mailx  


    # Create fresh keys

    mkdir -p /etc/openvpn/easy-rsa/

    cp /usr/share/openvpn/easy-rsa/2.0/* /etc/openvpn/easy-rsa/

    cd /etc/openvpn/easy-rsa/

    source vars
    export KEY_COUNTRY=”US”
    export KEY_PROVINCE=”CA”
    export KEY_CITY=”test-city”
    export KEY_ORG=”Example Company”
    export KEY_EMAIL=”royans@example.com”
    export KEY_CN=changemenow
    export KEY_NAME=changemenow
    export KEY_OU=changemenow
    ./clean-all
    ./build-dh
    ./pkitool –initca
    ./pkitool –server server
    ./build-key-pkcs12 –pkcs12 hostname 
    # Send a copy of ca cert by mail
    mail -s ca.cert -a /etc/openvpn/easy-rsa/keys/ca.crt royans@example.com <
    This is the cert file for this setup. Install this in the authorities tab in the chrome os device from where vpn needs to be initiated.
    EOF 
    # Create a server.conf file
    cat > /etc/openvpn/server.conf <
    port 1194
    proto udp
    dev tun
    ca /etc/openvpn/easy-rsa/keys/ca.crt
    cert /etc/openvpn/easy-rsa/keys/server.crt
    key /etc/openvpn/easy-rsa/keys/server.key  
    # This file should be kept secret
    dh /etc/openvpn/easy-rsa/keys/dh1024.pem
    server 10.8.0.0 255.255.255.0
    ifconfig-pool-persist ipp.txt
    keepalive 10 120
    persist-key
    persist-tun
    status openvpn-status.log/proc/sys/net/ipv4/ip_forward 
    verb 6
    plugin /usr/lib/openvpn/plugin/lib/openvpn-auth-pam.so login
    client-cert-not-required
    push “dhcp-option DNS 8.8.8.8”
    push “dhcp-option DNS 8.8.4.4”
    EOF  
    #Start the service
    /etc/init.d/openvpn start

      Step 6 – Setup basic source nat

      echo 1 > /proc/sys/net/ipv4/ip_forward 

      IP=`ifconfig eth0 | grep ‘inet addr’ | awk ‘{print $2}’ | cut -d’:’ -f2`  

      iptables -F; iptables -t nat -A POSTROUTING -o eth0 -j SNAT –to $IP

      Step 7 – Install cert

      • In step 5 we sent the ca.cert to the user’s email address. The user should download that cert and install it under authorities tab of chromebook [ chrome://chrome/settings/certificates#cert ]
      • There is a known issue that new certs of this type don’t take effect until user logs out and logs in again
      • After logging in again add the new vpn
        • chrome://chrome/settings/
        • Click on “Add connection”
        • Click on “Add private network”
        • In “Server hostname:” put in the external IP address or the name of your EC2 instance. For example ec2-54-245-135-132.compute-1.amazonaws.com
        • in “Server CA certificate” you should see a new certificate called “chagemenow”
        • For Username/Password use the credentials you setup in step 4

      Step 8 – Done

        • You should be able to test by pinging www.google.com from the crosh terminal

        Certificate based authentication

          • If your goal is to setup certificate based authentication you will have to do a few extra steps
            • Along with installing ca.cert in authorities tab, you would also have to install  the hostname.p12 certificate you created in “Step 5” onto your chromebook ( look in the keys directory )
            • In “server.conf” file comment out  both of the  lines below. The first one disables PAM based authentication, and second one enables certificate based authentication.
              • plugin /usr/lib/openvpn/plugin/lib/openvpn-auth-pam.so login
              • client-cert-not-required
            • When you specify client configuration make sure you specify both “Server CA certificate” and “User certificate”.
            • Type something into username to keep the UI satisfied… if you haven’t enabled PAM based authentication it would have no effect in the login process.

          Next steps

          • Note that this is the simplest Openvpn setup. There are many different ways you can improve on this, which I highly recommend if you plan to keep this instance running for more than a few minutes or hours.
            • You could use client certificates instead of just username/password
            • You should consider tightening firewall rules on the server  
            • If openvpn is running as root on the server, please switch it to something less privileged like ‘nobody’.
          • You should replace passwords and names I used as example above.
          • It should be trivial for someone to write a script to do this… if you do, please let me know and I’ll gladly link it from here
          • While EC2 instances costs only a few cents per hour… please do remember to shutdown when you are done, else you will get an unexpected bill at the end of the month.

          How software defined radios (SDRs) will change security

          Locks were considered very secure until the first lock pickers got their hands on it. Phone system were secure until the John Draper discovered some use for the toy whistle in Captain Crunch pack. Infact even the creators of internet didn’t think of too much security when it was initially designed.  Its the commoditization of technology which sometimes brings about the worst of all security bugs. And I believe the next round of changes are coming very soon.

          Until very recently radios were built for a purpose and they rarely did more than what it was supposed to do.   Think of them like the early computers which took a whole room and could only do only type of a job per computer. Todays computer can do all kinds of stuff and unlike the older versions, they don’t need to be rewired physically to make them do a new job. Everything is done using software.

          Wikipedia does a good job at defining what this is.

          software-defined radio system, or SDR, is a radio communication system where components that have been typically implemented in hardware (e.g. mixers,filtersamplifiersmodulators/demodulatorsdetectors, etc.) are instead implemented by means of software on a personal computer or embedded system.[1] While the concept of SDR is not new, the rapidly evolving capabilities of digital electronics render practical many processes which used to be only theoretically possible.

          A group of individuals figured out that some of the TV tuner cards can not only be reprogrammed to listen to a wider range of frequencies but could be driven entirely using software which could make it look like an all purpose radio receiver. Interestingly that USB tuner costs only about USD 20.

          PaulDotCom mentioned SDRs in one of the talks as well but he went further and pointed out that SDRs could also be used to send signals which makes it significantly more dangerous. One of the worst examples he gave was that an SDR could be reprogrammed to generate fake transponder signals. They pointed out that modern aircrafts do listen for transponder signals from other nearby aircrafts and some of them are programmed to take automatic sudden evasive measures when it detects another aircraft close by.

          The point is not that terrorists can attack airplanes this way… they could do it today by buying and reprogramming a real transponder. The point is that this technology will become so cheap that anyone would be able to do it with just a computer and a simple SDR transmitter.

          I’m not really sure how good Transponders are with respect to security.. may be it has a good secure way of authenticating the transmitter. In which case all is good. But if thats not happening today, it will change at some point when this technology becomes as easy to disrupt as DNS is today.

          Chrome: Fully sandboxed flash engine protect users

          The truth is that not everyone gets updates to chrome as soon as its released. And as its usually the case a lot of holes get discovered only after its exploited in the field. Google has finally announced a fully sandboxed flash engine which prevents the malicious code running within the flash component to fully exploit the system. It should keep you safe from unexpected security threats until an update arrives.

          Google says sandboxing is now available for Flash “with this release” of Chrome. The most recent version, Chrome 23, arrived last week, which is when the four-year-old browser received its usual dose of security fixes (14 in total), as well as a new version of Adobe Flash. 

          Yet the company today wanted to underline today that Chrome’s built-in Flash Player on Mac now uses a new plug-in architecture which runs Flash inside a sandbox that’s as strong as Chrome’s native sandbox, and “much more robust than anything else available.” This is great news for Mac users since Flash is so very widely used, and thus is a huge target for cybercriminals pushing malware. 

          Malware writers love exploiting Flash for the same reasons as they do Java: it’s a cross-platform plugin. Such an attack vector allows them to target more than one operating system, more than one browser, and thus more than one type of user. What Google is doing here is minimizing the chances that its users, namely those using Chrome, will get infected by such threats.