Thursday, August 11, 2016

Hurricane Earl update

[The following email and photographs were received today from Albert Jones (CCCCC), in response to a query about the effects of Hurricane Earl on the South Water Caye CREWS/CCCCC buoy.]
These are the first pictures I received of the South Water Caye buoy. I have requested from Sam to get some closer shots and to investigate if there were any salt water intrusion in the hull. Will share as soon as I get more info. So in a nutshell, both systems are on land at this time.
The SWBZ1 buoy as seen washed onto land, after Hurricane Earl, August 2016. Click on this image to see it in its original size.
The SWBZ1 buoy as seen washed onto land, after Hurricane Earl, August 2016. Click on this image to see it in its original size.
For purposes of comparison I am attaching an older photo of this buoy, showing its location on November 2nd, 2015. At the time I visited the buoy was anchored in a cove by a Belize Fisheries Rangers Station near South Water Caye, and I am assuming it remained in that location until impacted by the effects of Hurricane Earl.

-- Mike J+
The SWBZ1 buoy as I last saw it on November 2nd, 2015. Click on this image to see it in its original size.


Wednesday, January 13, 2016

AOMLers spend a few hours with the buoy

Sam Novello, Albert Jones and Mike Shoemaker working on the buoy (click to see a larger version).
On November 2nd, 2015, Albert Jones of CCCCC led AOMLers Mike Shoemaker and myself (Mike Jankulak) on an excursion to visit the South Water Caye buoy. We left Belize City in the morning, flew down to Dangriga, and were taken out to the Ranger Station by a small boat that I believe was operated by Fisheries. We spent maybe two or three hours with the buoy all told before returning to Dangriga by boat and then by plane to Belize City.

Our main purpose was to assess the buoy's readiness for redeployment and acquaint ourselves in person with the CREWS buoys' dual-datalogger design. Until this point we'd only had hands-on experience with the single-datalogger design which is unique to the Calabash Caye buoy. The seven other buoys in the CCCCC/CREWS network (South Water Caye, Little Cayman, two in Tobago, Barbados, two in the Dominican Republic) are all built with a dual-datalogger design that until now we'd only experienced via design schematics and source code reviews.

Going in we were aware of two serious issues: (1) the "Met" Junction Box had been repaired by YSI but after it was reinstalled on the buoy (late 2014?) it wasn't communicating with the rest of the buoy. (2) Although AOML had been connecting to the "Main" Junction Box via cellular service since April/2014 and downloading its diagnostics data, on June 25th, 2015 we'd mysteriously lost all contact with the buoy, suggesting a possible power failure. In addition to troubleshooting these two issues, Albert Jones had a third objective for this trip which was to install the EXO Water Quality Sonde that he'd carried with him from Belize City and to confirm that it was working correctly.

Following our visit, the buoy was left powered on, communications between the Main and Met dataloggers had been restored, and AOML's systems were once again receiving data reports via cellular service.  The EXO was installed and at least its sea temperature and salinity parameters seemed to be reporting reasonable values.  On the downside, the Vaisala WXT at this site is nonfunctional and probably irreparable, the solar regular's low voltage disconnect (LVD) was tripped and not yet reset, and the Compass is reporting garbage data. We did not have enough time to open the Met Junction Box and look inside but even if we had, there weren't any local facilities that were appropriate for this task (i.e. good lighting, large work surfaces, access to power outlets, internet connectivity, etc.).

Some detailed comments follow in point form:
  • We began by trying to power on the buoy's systems.  The solar panels were working and the batteries were fully charged.  However, during our diagnostics we found that the solar panels were suddenly failing to connect to anything.  This was fixed by unplugging and re-plugging the cable that connects to the Solar Panels Junction Box, so there may be some loose connection(s) relating to the solar panels, possibly in the "hot" connection to the underside of the solar panels connector through the Main JB lid.
A view of an ASC solar regulator (click to see a larger version).
  • Eventually we figured out that the ASC Solar Regulator's Low Voltage Disconnect (LVD) had tripped but, without any documentation for the ASC or access to the internet, we had no way of researching how to reset it. Instead we simply rewired the system to bypass this protection. I am told that this same wiring bypass was necessary at the Buccoo Reef (BUTO1) buoy in Tobago, and that the Buccoo ASC has resisted all efforts to reset its LVD. There have also been reports from the Little Cayman (CCMI2) buoy about power losses traced to the LVD, and as of this writing (January 2016) both of the newly-deployed Dominican Republic buoys (CWDR1 and PPDR1) have lost power only three weeks into their deployments (one after experiencing intermittent power losses) so the solar regulator could be implicated in power losses at as many as five of the eight CREWS buoys.
  • When our attention turned to the problem of the communication failure between the Main and Met Junction Boxes, the cause was immediately obvious: they were not in any way connected. The main 12-conductor "umbilical" cable that should run between the two JBs was not present.  The bulkhead connector on the Main JB was dummy-plugged and protected from the elements. The connector on the Met JB was not plugged, although since the Met JB's connectors are all pointing downwards the unplugged connector at least was somewhat protected from weather.  But if there are more Main/Met connection problems at this station, replacing the umbilical's bulkhead connector through the Main JB lid would be a good first step.
  • The missing cable was found in one of the boxes in storage at the Ranger Station.  This might seem like a serious oversight, since how could the two JBs be expected to communicate if there was nothing connecting them?  However, consider that the Met JB was removed (by Jon and Albert, I am told) in April of 2014 and it was returned to Ranger Station personnel, I believe, roughly six months later. Upon its return, all of the Met sensors were properly connected to the Met JB. Since the Main JB connector was dummy-plugged and the umbilical cable in storage, there was no obvious cables or connectors unplugged (apart from the open connector on the underside of the Met JB).  Given that this station is not the day-to-day primary responsibility of those who reinstalled the Met JB it is not terribly surprising that they did not know about the missing cable in storage at the station.  These events simply reiterate the importance of in-depth and repeated training of our local collaborators as well as the need for expert oversight from CCCCC, NOAA or AOML during more complicated operations like installations or swapouts.
  • We learned that the Vaisala WXT (an integrated "weather transmitter" with acoustic wind/precipitation sensors as well as internal air temperature/humidity/pressure sensors) had fallen off the station sometime during the summer of 2015 and was recovered from the salt water. We connected to it to see if it would respond but it was completely dead.  It is not expected that this WXT is reparable, not by us and not by Vaisala, after its salt-water exposure.
  • It's worth noting that the WXT mount seems problematic at half of the CCCCC/CREWS sites, and maybe more.  At South Water the WXT fell off.  At both Tobago stations "the bit where it was secured to the buoy seemed lose or easily slackened, and it was clearly off alignment" during the December/2015 maintenance visits (leading to a lifelong wind direction offset of the WXT that was later corrected manually in our data archives).  At the new Puerto Plata station in the Dominican Republic the WXT wind directions agreed to within 2° with the anemometer directions for the first 12 days of deployment, then slipped to an 8° difference for four days, then to a 122° difference for the station's final three days before losing power.  At CREWS pylons it was my practice to wrap the mount in self-amalgamating tape, then electrical tape, and sometimes to secure those with a cable tie, and I would recommend the same going forward for all CCCCC/CREWS buoys.
  • We briefly considered the hypothesis that it was the Vaisala's loss and the subsequent exposure of its cable end to rain and elements that might have caused a surge or short in the power systems that tripped the solar regulator's LVD. However, given that the Met canister was in no way connected to the Main JB, the batteries or the solar panels, it is impossible that a short in the WXT instrument or cable could have had any effect on the solar regulator. 
  • Note that I taped up the end of the WXT cable to avoid any moisture-related shorts, now that the cable is connected to a powered Met JB.  However we should consider replacing the WXT cable or at least the instrument-side connector.
  • The station was moored in shallow water near the Ranger Station dock, and not on dry land as we AOMLers had been told. It was somewhat awkward to lift the lid off the Main JB and access the station electronics while reaching over from a boat or while perched on the edge of the bouncing buoy, but we managed alright. I would recommend that the buoy be extracted from the water completely before our pre-deployment visit because we will need to climb the tower to install a new Vaisala WXT.
  • We were told that the station had originally been moored on the other side of the caye, and then sometime in the summer of 2015 it was towed to where we encountered it, in the shallow water near the dock of the ranger station.  All of these events -- the moving of the station, the loss of the Vaisala WXT, and the station's loss of power -- are believed to have been at about the same time, but it's not known where any one of these events may have directly caused another. We were also told that it was unusually rainy during this time (June of 2015).
  • I initially connected to the station by radio (RF401), using a unit I had brought with me. To my knowledge there was no RF transceiver, power adapter, or antenna/cable present at the ranger station. We did see more than one USB-to-serial adapter present. It may be advisable to conduct an inventory of the station's on-land buoy-related supplies and determine what equipment may be in need of replacement, if missing.
  • I was able to locate the station's maintenance cable (on one side it has a serial connector and on the other a four-pin connector that plugs directly into the outside of the Main JB, into a connector that is dummy-plugged during deployments).  I later connected to the station via this cable and confirmed that it is working properly.
  • Some of the EXO equipment is missing from the ranger station storage. There is no USB adapter or cable. They do have the magnetic "keys" that can be used to wake up the EXO so that it can be reached by bluetooth but since those keys are a different design than the one sent to AOML I did not initially recognize them.  Albert believed the EXO's D-cell batteries had been supplied and installed by YSI before they returned it but upon examination we found that there were no batteries present anywhere at the station, and certainly not inside the EXO. Whenever the EXO had last been removed from the buoy (uncertain), its cable had merely been coiled on top of the buoy without any dummy plug, so the EXO-side connector has been exposed to weather and salt spray for many months and is visibly corroded. At first I wasn't certain we could test or connect the EXO at all, without any way of supplying power or waking its bluetooth connection. However I eventually realized I could power the EXO off the buoy itself (using the corroded cable) and the station personnel found the magnetic key, so I connected the EXO to the buoy and put it into deployment mode, then confirmed that it was reporting data to the buoy. While on site only the sea temperatures seemed to be reasonable but in the days after our visit the salinities seemed to self-adjust to reasonable levels as well.  The other parameters (turbidity, fDOM, algae) seem far less certain, reporting either all zereos or a mixture of positive/negative values.  NO ON-SITE CALIBRATIONS were performed because Albert had been assured by YSI that all of the modular sensors had been newly-calibrated by YSI.  No attempt was made to wrap any of the sensors (or the EXO itself) in anti-fouling tape.
  • As of January 2016, the EXO has now been deployed (in the water, in the shallows near the ranger station dock) for more than two months.  It has no batteries.  It has no anti-fouling tape.  Its sea temperatures and salinities seem reasonable but after two months it is surely due for another on-site user recalibration. Its cable has a corroded connector on the EXO side.
  • We also brought with us a new cover for the EXO deployment tube.  This is a yellow plastic cylinder that fits over the top of the EXO tube and has a hinged lid that can be locked with a padlock. However, at the Calabash Caye site this part had been notched on the cylinder side to allow the EXO cable to emerge and connect to the buoy even while the lid was closed; the part sent for South Water Caye had no such notch.  Also it wasn't immediately clear (to me, at least) how the cover should be affixed to the tube -- possibly with glue, or screws? As far as I recall we left this new cover in storage at the ranger station and whatever old cover had been present was left untouched on the buoy. Finally, I don't think the ranger station has a padlock for this lid, even if the lid is successfully replaced at some later date.
  • During testing I found that the Met JB's internal compass was reporting negative numbers, which is out of spec. I do not know if the compass was tested and calibrated while the Met JB was being repaired at YSI. This could indicate a loose wire inside the Met JB or it could indicate that the compass is damaged, possibly by the moisture that entered the Met JB before its repair.  I am recommending that during the next visit we bring a spare compass with us in case we need to swap out a bad compass.
  • Jim Hendee has given approval for AOML to donate one of our spare Vaisala WXTs to replace the destroyed WXT at South Water Caye (via email dated November 3rd, 2015).
  • As far as I know there is no dedicated laptop for maintenance work at this site.  This means that our local collaborators cannot connect to the station by radio or direct cable to troubleshoot, which might be useful if for example we lose our cellular connection to the station and we'd like to test whether its systems have any power at all. More seriously, most of the EXO user calibration routines need to be performed every month or two, and this absolutely requires a laptop with the appropriate EXO software installed.  We did look to see if the storage boxes had the EXO software on a CD or USB key, and we didn't find anything.
That's a lot of detail, I know. But all of these points are necessary for consideration for when we plan this buoy's next repair/deployment visit.  At that time the buoy should be recovered to land and within easy access of a facility that has work tables, power outlets, good lighting and some kind of (whether wired or wifi) internet access.  Air conditioning would be nice but (as at Calabash Caye) not strictly necessary.

We are grateful to CCCCC's Albert Jones and the ranger station personnel, particularly Sam Novelo, for making all of the arrangements for this trip and for their kind hospitality and cooperation at South Water Caye itself.

(signed) Mike Jankulak

Friday, January 8, 2016

Reconstructed SWBZ1 History

EXO data from the SWBZ1 buoy, from September of 2013 to January of 2016.

I have spent some time going over what email and data records exist from the entire history of this buoy, in an effort to better understand:
  • what went wrong with the station in the past
  • what might still be going wrong with it
  • whether any past data might be scientifically useful
[Note that throughout this post I will use the names "canister," "Junction Box" and "JB" interchangeably throughout to refer to the two enclosures, designated either "Met" or "Main," each of which houses a datalogger.]

Here is my reconstructed timeline:
  • September 19th - 20th, 2013: This is when the station's data record begins, and judging from an email from Matt Previte to Jim Hendee, the actual deployment took place on Friday, September 20th. This email suggests that there were problems with the buoy's cellular service (provided by BTL, or Belize Telemedia Limited) but that some kind of workaround was allowing YSI systems at least to access the buoy's data in real time.  AOML did not have any access to data from this buoy until the following April.
  • October 23rd - November 14th, 2013: The buoy was completely powered down for these three weeks, judging by the jump in timestamps while data record numbers remain consecutive.  Also the compass directions appear to be slightly different before/after the data break. It is likely that the buoy was recovered from its deployment site and either powered down completely or at least disconnected from the Met JB.  This may have been part of an early attempt to resolve the issues relating to its cellular service.  After November 14th the data record resumes and is suggestive of a buoy reporting from its deployment site (through April of 2014, anyway).
  • January 7th - February 14th, 2014: While other parameters remain consistent with a buoy reporting from its deployment site, during this period there is no EXO connected.  Either it was removed temporarily for some reason or it (or its communications with the buoy) failed completely.
  • April 7th - 11th, 2014: The met/ocean data record ends on April 7th and the diagnostics-only data record begins on the 11th.  Email from Matt Previte of YSI indicates that Jon and Albert had visited the buoy in order to work on the cellular service problem and, in fact, they fixed that problem.  However they discovered "that the met canister was flawed at this site and was allowing moisture to get inside" and so it had to be removed and returned to YSI for evaluation and repair.  While the Met JB was gone the buoy's Main JB electronics and datalogger continued to operate and AOML's systems started calling in (using the IP address associated with its cellular account) to collect data for the first time.  Without the Met JB, however, only the Main datalogger's diagnostics tables were being populated (i.e. no met or ocean data of any kind). In fact this diagnostics data record will continue unbroken from April of 2014 to June of 2015, more than a year.
  • June 27th - July 1st, 2014: Met data records were being logged for just a few day but none of the meteorological measurements are reasonable. The EXO appeared to be connected. Voltage levels seemed to be on low side. Since this is a period for which we have uninterrupted diagnostics data from the Main JB in Belize (with no sign of unusually low voltages) it is reasonable to conclude that the Met JB was powered up in some completely different location, possibly within the YSI facilities as part of their evaluation or repair of the Met JB.
  • November 1st - December 2nd, 2014: Diagnostics records indicate that the main battery was not being fully charged during this month.  This is possibly suggestive of some kind of human intervention but it is not known exactly what -- it could indicate that the buoy's solar panels were shaded from full sunlight for some reason or that the cable to the Solar Panel JB was disconnected temporarily.  Given the timing of events before and after this month, my best guess is that YSI returned the repaired Met JB to Belize around this time, and perhaps there were some (unsuccessful) attempts to install it that impacted the charging of the battery.  Note that the Met JB reinstallation failed because the JB was only connected to the met and ocean sensors but its main "umbilical" cable to the Main JB was never attached.
  • June 25th, 2015: On this date the buoy (still reporting diagnostics, still without a connection to the Met JB, still without any met or ocean data reports) lost power. At about this time two other things happened: the WXT (integrated weather reporting system made by Vaisala) came loose from its mount and fell into the water, effectively destroying it. Also the buoy was physically moved from one side of the Caye to the other, where it would be closer to the dock and better sheltered from wind and seas. Later investigation suggests that the power failure was due to the solar regulator's Low Voltage Disconnect being tripped, which separates the battery from the payload (but not from the solar panels). Note that the power failure is unlikely to be related to the loss of the Vaisala WXT, because the Met and Main JBs were not connected in any way. The bare cable end left behind by the WXT had no source of power and could not possibly have caused any kind of short in the Main JB electronics.
  • November 2nd, 2015: Mikes Shoemaker and Jankulak (i.e., me) visited the buoy for a few hours. This will be described in more detail in another post but we identified the issue with the solar regulator (and worked around it by rewiring the buoy to circumvent the LVD protection), bringing the buoy back online and making it reachable by cellular service once more. We noted the destruction of the Vaisala WXT, and that the Met JB's internal compass appeared to be reporting garbage data.  We connected the EXO (though without any D-cell batteries installed) and left it reporting temperatures and salinities from the shallows where the buoy was moored.
  • December 27th, 2015: While awaiting further repairs and eventual deployment the buoy has continued to report from its shallow mooring, but starting in late December AOML experienced communication dropouts. These communication problems are suggestive of a problem with the connection between the two dataloggers (within the Met and Main JBs) and also a potential power/recharging issue. These issues will be analyzed more thoroughly for another post.
(signed) Mike Jankulak