March 12, 2014

Debugging user and device policies for Chrome OS

If you have used Chrome OS in a school or an enterprise network, you would have noticed how helpful the management piece can be. Using this tool you can quickly setup and deploy policies to make things easier for your users.

This is the authoritative source of all policies available on Chrome today. Pay special attention to the "Supported on" section. If it mentions "Google Chrome OS", then the policy is supported on devices and most of them can be set using Admin console UI.

There are essentially two different types of policies one can set on Chrome OS.

User policies

The "user policies" are those policies which can be set for an individual, regardless of which machine they are using Chrome from. A good example of a user policy is the "Screen Lock". An enterprise admin could enforce users to have an idle screen lock enabled automatically to protect internal company data. Similarly, there may be organizations which may want to disable "Browser History" across all users.

These "User policies"  will follow the user on all platforms, which means that in addition to working on Chrome OS these policies will also take effect on Chrome for Windows and Linux if the user signs into them with the organization's credentials.

Device policies

The "device policies" are policies applied to the machine irrespective of the user on it. For Chrome OS the policies which can be applied on the device are clearly defined in the policy list.

Examples of these policies are shown on the right. If a device is used in a lab environment which doesn't need data persistence, its simple to set "User Data" policy to "Erase all local user info, setting... after each sign-out". Note that these policies take effect only on devices which are enrolled into the domain.

Debugging

One of the first things an admin should learn is how to debug if the policies are setup correctly. The quickest way to do this is by going to the "chrome://policy" page. If there are any policies on your device, it will show up there.

There are two boxes on the top. The first one is labeled as  "Device policies" and the second one is labeled as "User policies". There are few different things you can quickly find out by looking at it:

  • The device is enrolled to "trialdevices.com"
    • If the "Device policies" box is missing, it probably means that the device is not enrolled.
  • The signed in user is rkt@blogofy.com
    • If the "User policies" box is missing, it probably means that the user is not part of a domain pushing policies.
  • Both policies were fetched in last 6 seconds (if this is too old, try to "Reload policies" to see if it can get a fresher version)
  • Status for both is "Policy cache OK"


If you notice stale policies, you should start investigating using a tool like this to see if there are firewalls in the way which could be impacting it. If that doesn't help, ask help from local networking admins who may know more.




Debugging user and device policies for Chrome OS

If you have used Chrome OS in a school or an enterprise network, you would have noticed how helpful the management piece can be. Using this tool you can quickly setup and deploy policies to make things easier for your users.

This is the authoritative source of all policies available on Chrome today. Pay special attention to the "Supported on" section. If it mentions "Google Chrome OS", then the policy is supported on devices and most of them can be set using Admin console UI.

There are essentially two different types of policies one can set on Chrome OS.

User policies

The "user policies" are those policies which can be set for an individual, regardless of which machine they are using Chrome from. A good example of a user policy is the "Screen Lock". An enterprise admin could enforce users to have an idle screen lock enabled automatically to protect internal company data. Similarly, there may be organizations which may want to disable "Browser History" across all users.

These "User policies"  will follow the user on all platforms, which means that in addition to working on Chrome OS these policies will also take effect on Chrome for Windows and Linux if the user signs into them with the organization's credentials.

Device policies

The "device policies" are policies applied to the machine irrespective of the user on it. For Chrome OS the policies which can be applied on the device are clearly defined in the policy list.

Examples of these policies are shown on the right. If a device is used in a lab environment which doesn't need data persistence, its simple to set "User Data" policy to "Erase all local user info, setting... after each sign-out". Note that these policies take effect only on devices which are enrolled into the domain.

Debugging

One of the first things an admin should learn is how to debug if the policies are setup correctly. The quickest way to do this is by going to the "chrome://policy" page. If there are any policies on your device, it will show up there.

There are two boxes on the top. The first one is labeled as  "Device policies" and the second one is labeled as "User policies". There are few different things you can quickly find out by looking at it:

  • The device is enrolled to "trialdevices.com"
    • If the "Device policies" box is missing, it probably means that the device is not enrolled.
  • The signed in user is rkt@blogofy.com
    • If the "User policies" box is missing, it probably means that the user is not part of a domain pushing policies.
  • Both policies were fetched in last 6 seconds (if this is too old, try to "Reload policies" to see if it can get a fresher version)
  • Status for both is "Policy cache OK"


If you notice stale policies, you should start investigating using a tool like this to see if there are firewalls in the way which could be impacting it. If that doesn't help, ask help from local networking admins who may know more.




March 04, 2014

Getting better over time : Troubleshooting Chrome OS updates

One of the salient features of Chrome OS is its ability to do transparent updates with little or no interaction from the user. This not only ensures the user is always protected, it also improves performance and features over time.


If you administer a fleet of chrome devices, I recommend you read this support on how to correctly configure this. It goes through all the autoupdate (AU) terminologies and suggests best practices which will help you in long run.

How to check if AU is properly working in your network

  1. I'll strongly recommend to enable reporting using your admin console. This will allow devices to send its OS version to the reporting engine.
  2. Then use the Admin SDK APIs to generate reports for devices in your domain, grouped by major version. I recommend you write your own scripts, but you can try third party tools like these to understand what the APIs are capable of.
  3. If you have lots of different networks, try to generate separate reports for devices in different networks to see if any of them are further behind than others. 
  4. If you do notice some devices are not getting updates, look for the following failure patterns
    1. Check if the devices are being blocked from reaching update engine. White-list these domains if you need to implement ACLs.
    2. If you have enabled "scattered" updates, disable it for those users. "Scattered" updates is not good for devices which are not used every single day.
    3. If your devices never get updates, you may have to reach out to Google's enterprise support team for deeper analysis.

    How to check what my devices should be at ?

    Google publishes all the version numbers for all of its chrome devices at this location:  http://cros-omahaproxy.appspot.com/. If you know what hardware you have, you should be able to find the latest stable, beta and dev version numbers there. Here is what it looks like for CR-48s and original Samsung chromebooks.

    Understanding AU Logs

    1. You can extract the AU logs from the device very easily using the following steps
      1. Go to chrome://net-internals#chromeos
      2. Click on "store debug logs". This creates a compressed archive in the "Download folder"
    2. After exploding the compressed archive, look for files under "/var/log/update_engine" which has all the AU related logs
    3. Find the latest log file there and open it up in an editor
    4. The AU request would look something like this
      <request protocol="3.0" version="ChromeOSUpdateEngine-0.1.0.0" updaterversion="ChromeOSUpdateEngine-0.1.0.0" installsource="scheduler" ismachine="1">
          <os version="Indy" platform="Chrome OS" sp="4319.96.0_i686"></os>
          <app appid="{9D137383-EB72-4BA9-A523-91AC0853F8AD}" version="4319.96.0" track="stable-channel" lang="en-US" board="parrot-signed-mp-v3keys" hardware_class="PARROT RAIL A-D 3974" delta_okay="true" fw_version="Google_Parrot.2685.37.0" ec_version="00BE107A00" >
              <updatecheck targetversionprefix=""></updatecheck>
              <event eventtype="3" eventresult="2" previousversion=""></event>
          </app>
      </request>
    5. Interpretation of the request
      • Current version of the OS: 4319.96.0_i686
      • Hardware is : PARROT (find more info here)
      • Lang="en-US" means its a US version of hardware
      • track="stable-channel" means this is requested a stable update 
      • Lang="en-US" means its a US version of hardware
      • track="stable-channel" means its requesting a stable update 
    6. The AU response would look something like this
      [1121/083519:INFO:omaha_request_action.cc(717)] Omaha request response: <?xml version="1.0" encoding="UTF-8"?><response protocol="3.0" server="prod"><daystart elapsed_days="2516" elapsed_seconds="23720"/><app appid="{9D137383-EB72-4BA9-A523-91AC0853F8AD}" status="ok"><updatecheck status="noupdate"/><event status="ok"/></app></response>
    7. Interpretation of response
      status="noupdate" means there are no updates available for this particular device
    8. If your device is getting "noupdate" even though its supposed to be on a newer version, something may be wrong. If this goes on for a few days, contact support to troubleshoot further.




    Getting better over time : Troubleshooting Chrome OS updates

    One of the salient features of Chrome OS is its ability to do transparent updates with little or no interaction from the user. This not only ensures the user is always protected, it also improves performance and features over time.

    https://youtube.googleapis.com/v/QtavfiBOv_g&source=uds

    If you administer a fleet of chrome devices, I recommend you read this support on how to correctly configure this. It goes through all the autoupdate (AU) terminologies and suggests best practices which will help you in long run.

    How to check if AU is properly working in your network

    1. I'll strongly recommend to enable reporting using your admin console. This will allow devices to send its OS version to the reporting engine.
    2. Then use the Admin SDK APIs to generate reports for devices in your domain, grouped by major version. I recommend you write your own scripts, but you can try third party tools like these to understand what the APIs are capable of.
    3. If you have lots of different networks, try to generate separate reports for devices in different networks to see if any of them are further behind than others. 
    4. If you do notice some devices are not getting updates, look for the following failure patterns
      1. Check if the devices are being blocked from reaching update engine. White-list these domains if you need to implement ACLs.
      2. If you have enabled "scattered" updates, disable it for those users. "Scattered" updates is not good for devices which are not used every single day.
      3. If your devices never get updates, you may have to reach out to Google's enterprise support team for deeper analysis.

      How to check what my devices should be at ?

      Google publishes all the version numbers for all of its chrome devices at this location:  http://cros-omahaproxy.appspot.com/. If you know what hardware you have, you should be able to find the latest stable, beta and dev version numbers there. Here is what it looks like for CR-48s and original Samsung chromebooks.

      Understanding AU Logs

      1. You can extract the AU logs from the device very easily using the following steps
        1. Go to chrome://net-internals#chromeos
        2. Click on "store debug logs". This creates a compressed archive in the "Download folder"
      2. After exploding the compressed archive, look for files under "/var/log/update_engine" which has all the AU related logs
      3. Find the latest log file there and open it up in an editor
      4. The AU request would look something like this
           
           
               
               
           
      5. Interpretation of the request
        • Current version of the OS: 4319.96.0_i686
        • Hardware is : PARROT (find more info here)
        • Lang="en-US" means its a US version of hardware
        • track="stable-channel" means this is requested a stable update 
        • Lang="en-US" means its a US version of hardware
        • track="stable-channel" means its requesting a stable update 
      6. The AU response would look something like this
        [1121/083519:INFO:omaha_request_action.cc(717)] Omaha request response:
      7. Interpretation of response
        status="noupdate" means there are no updates available for this particular device
      8. If your device is getting "noupdate" even though its supposed to be on a newer version, something may be wrong. If this goes on for a few days, contact support to troubleshoot further.




      March 03, 2014

      Putting Chrome OS behind SSL based webfilters

      Educational institutions, particularly the K12 are stuck between a rock and a hard place. They are always in search of ways to open up newer technologies to students, but don't want to give up their ability to manage and filter what students can see or do. As a father of two I approve that.

      While Chrome OS does take security very seriously and tries very hard to discourage "man in the middle attack", it does provide an industry tested feature to allow educators to filter web content for students in its recent version of Chrome OS. To understand how it works in Chrome OS, I'll first explain how the Chrome OS works internally.

      Chrome OS devices, as most of you already know, has two distinct components. The Chrome browser is what provides most of the UI, but deep inside it also has an operating system built on top of linux. Among other things that OS is responsible for, auto-updates and security are two of the most important. 

      The web filtering feature which Chrome OS provides for our enterprise and schools users allows all "user session" traffic from the browser to be intercepted, but doesn't allow any of the system requests to be intercepted in the same way.

      Network setup

      To get a chromebook to work correctly in an environment with webfilter, its important to let webfilter know which hosts chromebook would connect to for which it won't tolerate SSL inspection. Google has published a set of domain names here which can be used for this purpose.

      Note that whitelisting by IP addresses (netblocks) is not good enough. The IP addresses mapped to these hosts keep changing and the only reliable way to whitelist them is by whitelisting the domain names as it is. Most webfilters (including some transparent webfilters) support this and if you are not sure, contact your proxy/webfilter provider to understand how to do it.

      Quick test

      Once the network is setup, import your custom root CA cert into the browser using certificate
      manager under "Authorities" and make sure you enable "Trust this certificate for identifying websites." Then go to any website which you think should be intercepted and try to see if browser threw any error. Even if it didn't throw an error, check at the certificate details and confirm that it was signed by your webfilter.

      Broader test

      Once the tests confirm that everything is working as expected, its time to do a broader test using management console. To prepare for this test, I would recommend picking a small set of users who are are ok with brief interruption (in case something goes wrong) and are willing to provide you with detailed feedback to help you debug the issue.

      In your admin panel, go to Chrome's "Advance settings" section and then "Networks". Pick the OU where all of your test users are and then click on "Manage Certificates" button on the
      top right corner. 

      Upload your certificate and check the box for "Use the certificate as an HTTPS certificate authority". 

      The example on the right shows my setup where I'm using zscaler's cloud based webfilter.

      The final test to make sure this is good, would be to move these users to a network where there is no direct network access.  Have them be forced to go through the proxy/webfilter and see if anything breaks. 

      Let this configuration stay like this for a few days/weeks and collect feedback on whether users noticed any other side effects. For example make sure devices are getting updates (which is critical) and that user users can be added any time.

      Complete the transition

      Once everything has been tested, apply the certificate to more and more OUs until the transition is complete.

      Caveats

      There are few caveats you should watch out for
      1. Even though this policy is being applied as a user policy, it will only work on devices which are enrolled to the same domain. This is one of the most common reasons for the feature not working.  This also means that if the device was unenrolled, it may cause network connectivity failures.
      2. Since this is a user policy, other users using the same device will not get this feature automatically. Each user has to be moved into an OU where this certificate is installed.

      More info

      Putting Chrome OS behind SSL based webfilters

      Educational institutions, particularly the K12 are stuck between a rock and a hard place. They are always in search of ways to open up newer technologies to students, but don't want to give up their ability to manage and filter what students can see or do. As a father of two I approve that.

      While Chrome OS does take security very seriously and tries very hard to discourage "man in the middle attack", it does provide an industry tested feature to allow educators to filter web content for students in its recent version of Chrome OS. To understand how it works in Chrome OS, I'll first explain how the Chrome OS works internally.

      Chrome OS devices, as most of you already know, has two distinct components. The Chrome browser is what provides most of the UI, but deep inside it also has an operating system built on top of linux. Among other things that OS is responsible for, auto-updates and security are two of the most important. 

      The web filtering feature which Chrome OS provides for our enterprise and schools users allows all "user session" traffic from the browser to be intercepted, but doesn't allow any of the system requests to be intercepted in the same way.

      Network setup

      To get a chromebook to work correctly in an environment with webfilter, its important to let webfilter know which hosts chromebook would connect to for which it won't tolerate SSL inspection. Google has published a set of domain names here which can be used for this purpose.

      Note that whitelisting by IP addresses (netblocks) is not good enough. The IP addresses mapped to these hosts keep changing and the only reliable way to whitelist them is by whitelisting the domain names as it is. Most webfilters (including some transparent webfilters) support this and if you are not sure, contact your proxy/webfilter provider to understand how to do it.

      Quick test

      Once the network is setup, import your custom root CA cert into the browser using certificate
      manager under "Authorities" and make sure you enable "Trust this certificate for identifying websites." Then go to any website which you think should be intercepted and try to see if browser threw any error. Even if it didn't throw an error, check at the certificate details and confirm that it was signed by your webfilter.

      Broader test

      Once the tests confirm that everything is working as expected, its time to do a broader test using management console. To prepare for this test, I would recommend picking a small set of users who are are ok with brief interruption (in case something goes wrong) and are willing to provide you with detailed feedback to help you debug the issue.

      In your admin panel, go to Chrome's "Advance settings" section and then "Networks". Pick the OU where all of your test users are and then click on "Manage Certificates" button on the
      top right corner. 

      Upload your certificate and check the box for "Use the certificate as an HTTPS certificate authority". 

      The example on the right shows my setup where I'm using zscaler's cloud based webfilter.

      The final test to make sure this is good, would be to move these users to a network where there is no direct network access.  Have them be forced to go through the proxy/webfilter and see if anything breaks. 

      Let this configuration stay like this for a few days/weeks and collect feedback on whether users noticed any other side effects. For example make sure devices are getting updates (which is critical) and that user users can be added any time.

      Complete the transition

      Once everything has been tested, apply the certificate to more and more OUs until the transition is complete.

      Caveats

      There are few caveats you should watch out for
      1. Even though this policy is being applied as a user policy, it will only work on devices which are enrolled to the same domain. This is one of the most common reasons for the feature not working.  This also means that if the device was unenrolled, it may cause network connectivity failures.
      2. Since this is a user policy, other users using the same device will not get this feature automatically. Each user has to be moved into an OU where this certificate is installed.

      More info