AOMLers spend a few hours with the buoy

Sam Novello, Albert Jones and Mike Shoemaker working on the buoy (click to see a larger version).

On November 2nd, 2015, Albert Jones of CCCCC led AOMLers Mike Shoemaker and myself (Mike Jankulak) on an excursion to visit the South Water Caye buoy. We left Belize City in the morning, flew down to Dangriga, and were taken out to the Ranger Station by a small boat that I believe was operated by Fisheries. We spent maybe two or three hours with the buoy all told before returning to Dangriga by boat and then by plane to Belize City.

Our main purpose was to assess the buoy's readiness for redeployment and acquaint ourselves in person with the CREWS buoys' dual-datalogger design. Until this point we'd only had hands-on experience with the single-datalogger design which is unique to the Calabash Caye buoy. The seven other buoys in the CCCCC/CREWS network (South Water Caye, Little Cayman, two in Tobago, Barbados, two in the Dominican Republic) are all built with a dual-datalogger design that until now we'd only experienced via design schematics and source code reviews.

Going in we were aware of two serious issues: (1) the "Met" Junction Box had been repaired by YSI but after it was reinstalled on the buoy (late 2014?) it wasn't communicating with the rest of the buoy. (2) Although AOML had been connecting to the "Main" Junction Box via cellular service since April/2014 and downloading its diagnostics data, on June 25th, 2015 we'd mysteriously lost all contact with the buoy, suggesting a possible power failure. In addition to troubleshooting these two issues, Albert Jones had a third objective for this trip which was to install the EXO Water Quality Sonde that he'd carried with him from Belize City and to confirm that it was working correctly.

Following our visit, the buoy was left powered on, communications between the Main and Met dataloggers had been restored, and AOML's systems were once again receiving data reports via cellular service.  The EXO was installed and at least its sea temperature and salinity parameters seemed to be reporting reasonable values.  On the downside, the Vaisala WXT at this site is nonfunctional and probably irreparable, the solar regular's low voltage disconnect (LVD) was tripped and not yet reset, and the Compass is reporting garbage data. We did not have enough time to open the Met Junction Box and look inside but even if we had, there weren't any local facilities that were appropriate for this task (i.e. good lighting, large work surfaces, access to power outlets, internet connectivity, etc.).

Some detailed comments follow in point form:

• We began by trying to power on the buoy's systems.  The solar panels were working and the batteries were fully charged.  However, during our diagnostics we found that the solar panels were suddenly failing to connect to anything.  This was fixed by unplugging and re-plugging the cable that connects to the Solar Panels Junction Box, so there may be some loose connection(s) relating to the solar panels, possibly in the "hot" connection to the underside of the solar panels connector through the Main JB lid.

A view of an ASC solar regulator (click to see a larger version).

• Eventually we figured out that the ASC Solar Regulator's Low Voltage Disconnect (LVD) had tripped but, without any documentation for the ASC or access to the internet, we had no way of researching how to reset it. Instead we simply rewired the system to bypass this protection. I am told that this same wiring bypass was necessary at the Buccoo Reef (BUTO1) buoy in Tobago, and that the Buccoo ASC has resisted all efforts to reset its LVD. There have also been reports from the Little Cayman (CCMI2) buoy about power losses traced to the LVD, and as of this writing (January 2016) both of the newly-deployed Dominican Republic buoys (CWDR1 and PPDR1) have lost power only three weeks into their deployments (one after experiencing intermittent power losses) so the solar regulator could be implicated in power losses at as many as five of the eight CREWS buoys.
• When our attention turned to the problem of the communication failure between the Main and Met Junction Boxes, the cause was immediately obvious: they were not in any way connected. The main 12-conductor "umbilical" cable that should run between the two JBs was not present.  The bulkhead connector on the Main JB was dummy-plugged and protected from the elements. The connector on the Met JB was not plugged, although since the Met JB's connectors are all pointing downwards the unplugged connector at least was somewhat protected from weather.  But if there are more Main/Met connection problems at this station, replacing the umbilical's bulkhead connector through the Main JB lid would be a good first step.
• The missing cable was found in one of the boxes in storage at the Ranger Station.  This might seem like a serious oversight, since how could the two JBs be expected to communicate if there was nothing connecting them?  However, consider that the Met JB was removed (by Jon and Albert, I am told) in April of 2014 and it was returned to Ranger Station personnel, I believe, roughly six months later. Upon its return, all of the Met sensors were properly connected to the Met JB. Since the Main JB connector was dummy-plugged and the umbilical cable in storage, there was no obvious cables or connectors unplugged (apart from the open connector on the underside of the Met JB).  Given that this station is not the day-to-day primary responsibility of those who reinstalled the Met JB it is not terribly surprising that they did not know about the missing cable in storage at the station.  These events simply reiterate the importance of in-depth and repeated training of our local collaborators as well as the need for expert oversight from CCCCC, NOAA or AOML during more complicated operations like installations or swapouts.
• We learned that the Vaisala WXT (an integrated "weather transmitter" with acoustic wind/precipitation sensors as well as internal air temperature/humidity/pressure sensors) had fallen off the station sometime during the summer of 2015 and was recovered from the salt water. We connected to it to see if it would respond but it was completely dead.  It is not expected that this WXT is reparable, not by us and not by Vaisala, after its salt-water exposure.
• It's worth noting that the WXT mount seems problematic at half of the CCCCC/CREWS sites, and maybe more.  At South Water the WXT fell off.  At both Tobago stations "the bit where it was secured to the buoy seemed lose or easily slackened, and it was clearly off alignment" during the December/2015 maintenance visits (leading to a lifelong wind direction offset of the WXT that was later corrected manually in our data archives).  At the new Puerto Plata station in the Dominican Republic the WXT wind directions agreed to within 2° with the anemometer directions for the first 12 days of deployment, then slipped to an 8° difference for four days, then to a 122° difference for the station's final three days before losing power.  At CREWS pylons it was my practice to wrap the mount in self-amalgamating tape, then electrical tape, and sometimes to secure those with a cable tie, and I would recommend the same going forward for all CCCCC/CREWS buoys.
• We briefly considered the hypothesis that it was the Vaisala's loss and the subsequent exposure of its cable end to rain and elements that might have caused a surge or short in the power systems that tripped the solar regulator's LVD. However, given that the Met canister was in no way connected to the Main JB, the batteries or the solar panels, it is impossible that a short in the WXT instrument or cable could have had any effect on the solar regulator. 
• Note that I taped up the end of the WXT cable to avoid any moisture-related shorts, now that the cable is connected to a powered Met JB.  However we should consider replacing the WXT cable or at least the instrument-side connector.
• The station was moored in shallow water near the Ranger Station dock, and not on dry land as we AOMLers had been told. It was somewhat awkward to lift the lid off the Main JB and access the station electronics while reaching over from a boat or while perched on the edge of the bouncing buoy, but we managed alright. I would recommend that the buoy be extracted from the water completely before our pre-deployment visit because we will need to climb the tower to install a new Vaisala WXT.
• We were told that the station had originally been moored on the other side of the caye, and then sometime in the summer of 2015 it was towed to where we encountered it, in the shallow water near the dock of the ranger station.  All of these events -- the moving of the station, the loss of the Vaisala WXT, and the station's loss of power -- are believed to have been at about the same time, but it's not known where any one of these events may have directly caused another. We were also told that it was unusually rainy during this time (June of 2015).
• I initially connected to the station by radio (RF401), using a unit I had brought with me. To my knowledge there was no RF transceiver, power adapter, or antenna/cable present at the ranger station. We did see more than one USB-to-serial adapter present. It may be advisable to conduct an inventory of the station's on-land buoy-related supplies and determine what equipment may be in need of replacement, if missing.
• I was able to locate the station's maintenance cable (on one side it has a serial connector and on the other a four-pin connector that plugs directly into the outside of the Main JB, into a connector that is dummy-plugged during deployments).  I later connected to the station via this cable and confirmed that it is working properly.
• Some of the EXO equipment is missing from the ranger station storage. There is no USB adapter or cable. They do have the magnetic "keys" that can be used to wake up the EXO so that it can be reached by bluetooth but since those keys are a different design than the one sent to AOML I did not initially recognize them.  Albert believed the EXO's D-cell batteries had been supplied and installed by YSI before they returned it but upon examination we found that there were no batteries present anywhere at the station, and certainly not inside the EXO. Whenever the EXO had last been removed from the buoy (uncertain), its cable had merely been coiled on top of the buoy without any dummy plug, so the EXO-side connector has been exposed to weather and salt spray for many months and is visibly corroded. At first I wasn't certain we could test or connect the EXO at all, without any way of supplying power or waking its bluetooth connection. However I eventually realized I could power the EXO off the buoy itself (using the corroded cable) and the station personnel found the magnetic key, so I connected the EXO to the buoy and put it into deployment mode, then confirmed that it was reporting data to the buoy. While on site only the sea temperatures seemed to be reasonable but in the days after our visit the salinities seemed to self-adjust to reasonable levels as well.  The other parameters (turbidity, fDOM, algae) seem far less certain, reporting either all zereos or a mixture of positive/negative values.  NO ON-SITE CALIBRATIONS were performed because Albert had been assured by YSI that all of the modular sensors had been newly-calibrated by YSI.  No attempt was made to wrap any of the sensors (or the EXO itself) in anti-fouling tape.
• As of January 2016, the EXO has now been deployed (in the water, in the shallows near the ranger station dock) for more than two months.  It has no batteries.  It has no anti-fouling tape.  Its sea temperatures and salinities seem reasonable but after two months it is surely due for another on-site user recalibration. Its cable has a corroded connector on the EXO side.
• We also brought with us a new cover for the EXO deployment tube.  This is a yellow plastic cylinder that fits over the top of the EXO tube and has a hinged lid that can be locked with a padlock. However, at the Calabash Caye site this part had been notched on the cylinder side to allow the EXO cable to emerge and connect to the buoy even while the lid was closed; the part sent for South Water Caye had no such notch.  Also it wasn't immediately clear (to me, at least) how the cover should be affixed to the tube -- possibly with glue, or screws? As far as I recall we left this new cover in storage at the ranger station and whatever old cover had been present was left untouched on the buoy. Finally, I don't think the ranger station has a padlock for this lid, even if the lid is successfully replaced at some later date.
• During testing I found that the Met JB's internal compass was reporting negative numbers, which is out of spec. I do not know if the compass was tested and calibrated while the Met JB was being repaired at YSI. This could indicate a loose wire inside the Met JB or it could indicate that the compass is damaged, possibly by the moisture that entered the Met JB before its repair.  I am recommending that during the next visit we bring a spare compass with us in case we need to swap out a bad compass.
• Jim Hendee has given approval for AOML to donate one of our spare Vaisala WXTs to replace the destroyed WXT at South Water Caye (via email dated November 3rd, 2015).
• As far as I know there is no dedicated laptop for maintenance work at this site.  This means that our local collaborators cannot connect to the station by radio or direct cable to troubleshoot, which might be useful if for example we lose our cellular connection to the station and we'd like to test whether its systems have any power at all. More seriously, most of the EXO user calibration routines need to be performed every month or two, and this absolutely requires a laptop with the appropriate EXO software installed.  We did look to see if the storage boxes had the EXO software on a CD or USB key, and we didn't find anything.

That's a lot of detail, I know. But all of these points are necessary for consideration for when we plan this buoy's next repair/deployment visit.  At that time the buoy should be recovered to land and within easy access of a facility that has work tables, power outlets, good lighting and some kind of (whether wired or wifi) internet access.  Air conditioning would be nice but (as at Calabash Caye) not strictly necessary.

We are grateful to CCCCC's Albert Jones and the ranger station personnel, particularly Sam Novelo, for making all of the arrangements for this trip and for their kind hospitality and cooperation at South Water Caye itself.

(signed) Mike Jankulak