The trouble with ubiquitous computing

The idea of “ubiquitous computing” most people dream about doesn’t usually include the troubles of patching them every week. It doesn’t even mention that there would be new bugs found daily and that most of the fixes would be available weeks if not months after they were discovered.

Windows XP

Windows XP has been in news recently because Microsoft has finally pulled support for this aging OS. 30% of all active desktops are still on XP and now we know of a new security bug, which would never get fixed for these users.

XP may eventually become the epitome of unpatched buggy software because of the visibility this issue got, but I feel this may just be the tip of the iceberg. For every XP out there, I bet there is one or more unpatched networking device just waiting for someone to exploit it, and this number is growing very fast. Some of these bugs are just that… bugs, but I suspect most of them are due to less then reputable code/design quality.  Its a wild-wild-west out there and this has to stop.

The other problem with ubiquitous computing is that the number of devices per house hold is growing rapidly and doing manual updates to every single one is getting close to impossible. We need to get to a place where users won’t have to worry about manually updating the devices. The industry as a whole needs to do a better job at promoting a type of automation and testing which requires significantly higher levels of investment in resources (by manufacturer) to make it happen. Apple with its iOS update infrastructure and Google with its Chrome updates has shown that its possible to do it at scale.

So what can we as users do ? For a start we may have an obligation to ask about auto-updates when we buy new devices. For connected devices at least, shipping updates shouldn’t be “optional”. Vote for the right manufacturer with your wallet.

 

 

Debugging user and device policies for Chrome OS

If you have used Chrome OS in a school or an enterprise network, you would have noticed how helpful the management piece can be. Using this tool you can quickly setup and deploy policies to make things easier for your users.

This is the authoritative source of all policies available on Chrome today. Pay special attention to the “Supported on” section. If it mentions “Google Chrome OS”, then the policy is supported on devices and most of them can be set using Admin console UI.

There are essentially two different types of policies one can set on Chrome OS.

User policies

The “user policies” are those policies which can be set for an individual, regardless of which machine they are using Chrome from. A good example of a user policy is the “Screen Lock”. An enterprise admin could enforce users to have an idle screen lock enabled automatically to protect internal company data. Similarly, there may be organizations which may want to disable “Browser History” across all users.

These “User policies”  will follow the user on all platforms, which means that in addition to working on Chrome OS these policies will also take effect on Chrome for Windows and Linux if the user signs into them with the organization’s credentials.

Device policies

The “device policies” are policies applied to the machine irrespective of the user on it. For Chrome OS the policies which can be applied on the device are clearly defined in the policy list.

Examples of these policies are shown on the right. If a device is used in a lab environment which doesn’t need data persistence, its simple to set “User Data” policy to “Erase all local user info, setting… after each sign-out”. Note that these policies take effect only on devices which are enrolled into the domain.

Debugging

One of the first things an admin should learn is how to debug if the policies are setup correctly. The quickest way to do this is by going to the “chrome://policy” page. If there are any policies on your device, it will show up there.

There are two boxes on the top. The first one is labeled as  “Device policies” and the second one is labeled as “User policies”. There are few different things you can quickly find out by looking at it:

  • The device is enrolled to “trialdevices.com”
    • If the “Device policies” box is missing, it probably means that the device is not enrolled.
  • The signed in user is rkt@blogofy.com
    • If the “User policies” box is missing, it probably means that the user is not part of a domain pushing policies.
  • Both policies were fetched in last 6 seconds (if this is too old, try to “Reload policies” to see if it can get a fresher version)
  • Status for both is “Policy cache OK”

If you notice stale policies, you should start investigating using a tool like this to see if there are firewalls in the way which could be impacting it. If that doesn’t help, ask help from local networking admins who may know more.




Getting better over time : Troubleshooting Chrome OS updates

One of the salient features of Chrome OS is its ability to do transparent updates with little or no interaction from the user. This not only ensures the user is always protected, it also improves performance and features over time.

https://youtube.googleapis.com/v/QtavfiBOv_g&source=uds

If you administer a fleet of chrome devices, I recommend you read this support on how to correctly configure this. It goes through all the autoupdate (AU) terminologies and suggests best practices which will help you in long run.

How to check if AU is properly working in your network

  1. I’ll strongly recommend to enable reporting using your admin console. This will allow devices to send its OS version to the reporting engine.
  2. Then use the Admin SDK APIs to generate reports for devices in your domain, grouped by major version. I recommend you write your own scripts, but you can try third party tools like these to understand what the APIs are capable of.
  3. If you have lots of different networks, try to generate separate reports for devices in different networks to see if any of them are further behind than others. 
  4. If you do notice some devices are not getting updates, look for the following failure patterns
    1. Check if the devices are being blocked from reaching update engine. White-list these domains if you need to implement ACLs.
    2. If you have enabled “scattered” updates, disable it for those users. “Scattered” updates is not good for devices which are not used every single day.
    3. If your devices never get updates, you may have to reach out to Google’s enterprise support team for deeper analysis.

    How to check what my devices should be at ?

    Google publishes all the version numbers for all of its chrome devices at this location:  http://cros-omahaproxy.appspot.com/. If you know what hardware you have, you should be able to find the latest stable, beta and dev version numbers there. Here is what it looks like for CR-48s and original Samsung chromebooks.

    Understanding AU Logs

    1. You can extract the AU logs from the device very easily using the following steps
      1. Go to chrome://net-internals#chromeos
      2. Click on “store debug logs”. This creates a compressed archive in the “Download folder”
    2. After exploding the compressed archive, look for files under “/var/log/update_engine” which has all the AU related logs
    3. Find the latest log file there and open it up in an editor
    4. The AU request would look something like this
         
         
             
             
         
    5. Interpretation of the request
      • Current version of the OS: 4319.96.0_i686
      • Hardware is : PARROT (find more info here)
      • Lang=”en-US” means its a US version of hardware
      • track=”stable-channel” means this is requested a stable update 
      • Lang=”en-US” means its a US version of hardware
      • track=”stable-channel” means its requesting a stable update 
    6. The AU response would look something like this
      [1121/083519:INFO:omaha_request_action.cc(717)] Omaha request response:
    7. Interpretation of response
      – status=”noupdate” means there are no updates available for this particular device
    8. If your device is getting “noupdate” even though its supposed to be on a newer version, something may be wrong. If this goes on for a few days, contact support to troubleshoot further.

    Putting Chrome OS behind SSL based webfilters

    Educational institutions, particularly the K12 are stuck between a rock and a hard place. They are always in search of ways to open up newer technologies to students, but don’t want to give up their ability to manage and filter what students can see or do. As a father of two I approve that.

    While Chrome OS does take security very seriously and tries very hard to discourage “man in the middle attack”, it does provide an industry tested feature to allow educators to filter web content for students in its recent version of Chrome OS. To understand how it works in Chrome OS, I’ll first explain how the Chrome OS works internally.

    Chrome OS devices, as most of you already know, has two distinct components. The Chrome browser is what provides most of the UI, but deep inside it also has an operating system built on top of linux. Among other things that OS is responsible for, auto-updates and security are two of the most important. 

    The web filtering feature which Chrome OS provides for our enterprise and schools users allows all “user session” traffic from the browser to be intercepted, but doesn’t allow any of the system requests to be intercepted in the same way.

    Network setup

    To get a chromebook to work correctly in an environment with webfilter, its important to let webfilter know which hosts chromebook would connect to for which it won’t tolerate SSL inspection. Google has published a set of domain names here which can be used for this purpose.

    Note that whitelisting by IP addresses (netblocks) is not good enough. The IP addresses mapped to these hosts keep changing and the only reliable way to whitelist them is by whitelisting the domain names as it is. Most webfilters (including some transparent webfilters) support this and if you are not sure, contact your proxy/webfilter provider to understand how to do it.

    Quick test

    Once the network is setup, import your custom root CA cert into the browser using certificate

    manager under “Authorities” and make sure you enable “Trust this certificate for identifying websites.” Then go to any website which you think should be intercepted and try to see if browser threw any error. Even if it didn’t throw an error, check at the certificate details and confirm that it was signed by your webfilter.

    Broader test

    Once the tests confirm that everything is working as expected, its time to do a broader test using management console. To prepare for this test, I would recommend picking a small set of users who are are ok with brief interruption (in case something goes wrong) and are willing to provide you with detailed feedback to help you debug the issue.

    In your admin panel, go to Chrome’s “Advance settings” section and then “Networks”. Pick the OU where all of your test users are and then click on “Manage Certificates” button on the

    top right corner. 


    Upload your certificate and check the box for “Use the certificate as an HTTPS certificate authority”. 

    The example on the right shows my setup where I’m using zscaler’s cloud based webfilter.

    The final test to make sure this is good, would be to move these users to a network where there is no direct network access.  Have them be forced to go through the proxy/webfilter and see if anything breaks. 

    Let this configuration stay like this for a few days/weeks and collect feedback on whether users noticed any other side effects. For example make sure devices are getting updates (which is critical) and that user users can be added any time.

    Complete the transition

    Once everything has been tested, apply the certificate to more and more OUs until the transition is complete.

    Caveats

    There are few caveats you should watch out for
    1. Even though this policy is being applied as a user policy, it will only work on devices which are enrolled to the same domain. This is one of the most common reasons for the feature not working.  This also means that if the device was unenrolled, it may cause network connectivity failures.
    2. Since this is a user policy, other users using the same device will not get this feature automatically. Each user has to be moved into an OU where this certificate is installed.

    More info

    Chrome device fleet reporting using APIs

    Chrome Devices have been a huge success in places like schools where students and teachers want the mobility and price point of a tablet but usability of a laptop. Like everything else Google focuses on scale and part of the Chrome Device offering to enterprises and schools are the tools around the devices to manage the fleet.
    Chrome Device management can be centralized which allows admins not only manage apps on the device, they can also push complex network and system settings with a touch of a few key strokes. And that works the same way for a customer with 10 devices and one with 20000.

    Recently Google released a new batch of APIs called the Admin SDK which includes some new APIs to discover and manage devices in the fleet. Using these APIs, admins can not only get a list of the active devices they have, they can also find out if the devices are having update issues. As an example of how to use this API, I’ve published a sample script called “ChromeOSDeviceStats” which I wrote for my own domain. This could be used to quickly get high level stats of devices in the domain, when they were enrolled, what version they are on, and what Orgs are they part off.

    The script could be further adapted to automatically send email reports like these when certain conditions are met. For example, if a school is interested in knowing about all recent “enrollments”, they could enhance this script to send that report.

    —————————————————————————
    Total devices : 31
    Total active : 31
    Total in Dev mode : 1
    Total devices enrolled in last 2 days : 0
    Total devices enrolled in last 7 days : 1
    Total devices enrolled in last 1 month : 2
    Total devices enrolled in last 1 year : 6
    Distribution of devices across models: 
                              Alex : 5 
                              Google_Stumpy : 2 
                              Google_Link : 6 
                              Google_Snow : 1 
                              Google_Lumpy : 8 
    Distribution of devices across versions: 
                              24 : 1 
                              25 : 5 
                              26 : 2 
                              27 : 5 
                              21 : 6 
                              23 : 3 
    Distribution of devices across orgUnits: 
                              /Eng : 3 
                              /hack : 1 
                              / : 31 
    —————————————————————————
     

    Openvpn in EC2 for Chromebooks : Part 2

    [ Update: take a look at this write up as well ]

    ChromeOS has a minimalistic design does a fairly good job at hiding the complex internals of the operating system. But deep inside it still runs linux and has a full blown openvpn client. In this post I’ll show you how to use “ONC” (Open Network Configuration) format to configure OpenVPN client inside ChromeOS. This file is very similar to an .ovpn file.

    The core scripts which did the cert conversion and created the sample ONC file was contributed by Ralph by reading the ONC documentations by himself. If there is a bleeding-edge user of chromebook.. he is the best example I can think off.

    These scripts are now published on github and here is a step by step guide of how to use it.

    Step 1: Launch a new amazon instance ( Based on Amazon Linux AMI )

    • Pick defaults for everything ( Use t1.micro for the cheapest instance )
    • If you don’t already have a keypair, create one yourself and upload your public key to amazon

    Step 2: Update security group you used. Allow UDP:1194 incoming.

    • Open 1194 UDP incoming 
    • Open 22 TCP incoming

    Step 3: Find the IP address of the new instance.

    • Find the “Public DNS” address. This is the address we will SSH to.

    Step 4: SSH into the server

    • If you only have a chromebook, use this extension to initiate ssh
    • Upload both private and public key to this extension.
    • Fill in the hostname and username (ec2-user)

    Step 5: Run the quick setup script

    curl https://raw.github.com/royans/ec2_chromeos_openvpn/master/quicksetup.sh > quicksetup.sh; 
    chmod +x quicksetup.sh;
    sudo ./quicksetup.sh email_address@blogofy.com
    • These three lines downloads the script and launches the setup script.
    • Remember to put in your email address instead of the one listed here
    • This step may take two to three minutes before it prompts you for anything

    Step 6: Select default values

    • When it does prompt you, just choose the defaults
    • If it asks you y/n questions… just select ‘y’ 
    • It will ask you for password a few times… just press enter (which sets no password on the keys)
    • Remember this is proof of concept and that you should customize it before you can use it for real stuff

    Step 7: Wait for the email. Download the ONC file attached.

    Step 8: Upload ONC file into chromebook

    • Make sure it says “ONC file successfully parsed” after the import.
    • [Advanced users: /var/logs/ui/ui.LATEST will have parse errors if you want to investigate a failed import]

    Step 9: Try to connect to the openvpn server

    • Just click on “Connect”

    Step 10: Connected

    • At this point you should see a solid (should not be blinking) VPN signal on the wifi icon. 

    Step 11: Verify if its traffic is being routed through amazon

    Patches: Pull or Push ?


    Most people prefer to disagree with the masses on whether they like sunny side up or scrambled eggs. And the form of getting patches is no different. If you ask an IT administrator (which is the person applying patches in most corporate organizations) they will tell you horror stories of how patches can go wrong and would be happy to give you examples of why every patch needs to be individually tested before deploying. 
    But my dad, for example, doesn’t care about patches, and while he won’t go out of the way to install a patch, he may be ok with patches being pushed to him automatically.  

    This debate  reminds me of another interesting debate in the Web-Operations world about “continuous deployment“. In that case the debate was whether applications should be deployed in scheduled releases (for example every quarter) or whether it should be released as things gets developed and pushed.

    If you think about this a little more it would be very clear that the developers who build the patches are the ones who first need to be ready to do “continuous deployment”. The confidence of every patch is directly proportional to the robustness of test infrastructure… and the unit-tests and integration-tests which are associated with it. For products where there are test-gaps it makes sense for IT administrators to constantly monitor and test every single patch before its deployed. 

    So that brings us back to “Pull or Push“. Because of the increase in number of recent attacks, I am now very conscious about what products I use on a daily basis and try very hard to pick those which can auto-update without nagging me. And they are usually the ones which have robust test infrastructure to allow “continuous deployment”… which in turn means that they are usually the ones which have better test coverage of their products and are the ones which can quickly patch something bad very quickly with confidence (like a 0-day).

    I do understand that mistakes in ‘Push’ based patches can be very expensive, but they are still more secure for end users when it comes to privacy and security.

    The reason why I was thinking about this is that Adobe released a patch today.. hope you noticed. I just wish it would auto-update devices where have it installed… its easy to forget to update devices, and in general not doing silent auto-update makes me worried that they are not super confident about their test infrastructure.

    Capturing wifi traffic of one station from another

    This is more of an embarrassing tale than a real how-to document. But I found this interesting enough that I don’t mind sharing it.

    A couple of weeks ago I was tasked to capture wifi traffic from a device which didn’t have any capture software built in and I wondered how one would do it.

    I have used sniffing tools on my Mac to passively sniff activity on access points around me. Because I’ve always tested such tools in places with dozens of access points with multiple saturated channels, I always assumed that all wifi stations ( laptops ) frequently switch channels. I also assumed that AP (Access points) which are setup to select channels automatically are designed to automatically switch channels anytime if they find a better (less noisy) frequency to provide services at.

    And because of those incorrect assumption, I concluded that sniffing another wifi station would be a difficult task because it would be impossible to dynamically change the channel of a second wifi station to follow the first one to correctly sniff all the packets.

    After a short discussion with a colleague I found out that most wifi stations don’t really switch AP points unless the noise to signal ratio gets too bad, and most APs never change channels once they are fully initialized.

    At the end, to sniff one device, all I had to do is keep the second device close to the first one and make sure that the second one joins the same channel as the first one. For my tests I used open wifi AP which were easier to capture/decode. At this point, if your hardware is capable of promiscuous mode and you have the right software for capture, u should be able to put in a filter with the mac address of the device you want to capture to initiate the process.

    Chrome Frame – How to add command line parameters

    Chrome frame intentionally does stuff without getting in the way of the user. This sometimes makes things harder to debug. For example how can one debug an issue if chrome frame doesn’t even launch ? Apparently there is a flag for that. But you have to know how to enable it. Here are the steps.

    1. Make sure chrome frame is installed.
    2. We can enable startup flags for dumping debug logs using a policy called AdditionalLaunchParameters
    3. If this is just for one desktop, I recommend doing a registry edit (it can be pushed via GPO as well)
    4. Add a REG_SZ property “AdditionalLaunchParameters” to “SoftwarePoliciesGoogleChromeAdditionalLaunchParameters” with the value “–enable-logging –v=1” (also documented here and mentioned here)  [ Attachment 1 ]
    5. Next kill the IE browser and make sure chrome is also dead by checking taskmgr
    6. Restart IE and go to “gcf:about:version” and confirm that the parameters you added show up next to “Command Line:”. If this doesn’t work… skip this step and go to next step anyway.
    7. Under your “Application Data” folder (its inside “Documents and Settings”) search for a file chrome_debug.log [ Its usually in “Application DataGoogleChrome FrameUser DataIEXPLORE