Complete Gantry -> Globus pipeline

@jdmaloney and I met today to discuss the remaining pieces of the puzzle for getting data from gantry -> globus -> NCSA. We were talking about who should do what and I wanted to start by writing things up. There are two main components:
1 MAC transfer initiator
This will track files coming from the gantry. It should monitor the place on disk where files are being sent, and periodically initiate globus transfers to NCSA with a cron job. This will need some lightweight status tracking to know if a file has been sent to globus yet, whether notification was sent to the NCSA API, etc. so that the same file isn't sent twice and files can be moved or archived after transmission.
When files are ready, this will initiate a globus transfer. This generates a globus ID, which is then sent to the NCSA API along with the initiating globus username for monitoring.
2 NCSA transfer monitor API
This piece will track files coming from globus. There is an API component that allows external users to notify when a new Globus transfer has started. It will monitor the place on disk where globus is writing files, and periodically check with the globus API for transfer status of known IDs. When a transfer is complete, this will notify clowder to index the file and write details to a log file.
The API has a config file where we list the trusted globus users who can submit jobs to it. Those users will send a globus task ID and their username to the API, which adds that task to the list of tasks that are checked each 1-2 seconds for status.  JD and I mostly discussed what 1 would look like:

identifying which files are completely transferred from the gantry and ready to go
way to track at what stage in prep are files coming off the gantry, e.g. a lightweight database/CSV/JSON file tracking whether upload to globus has begun, etc.
Max can write python code to initiate upload to Globus and notify 2's API that transfer has begun.
What else needs to be present?

I'm prepared to help JD with code to smooth out his end of the transfer, but wanted to create this issue so @robkooper and others could weigh in.

Thought about this last night, and I think we might be able to use a good chunk of my code for 2 with some modifications to address 1 as well.
2 has methods for tracking the status of file transfers including flags (i.e. IN-PROGRESS, FAILED, SUCCEEDED) and writing those to disk in a logfile every second. A similar process could monitor the directory structure where the gantry files are being stored, use JSON files to store file status along the workflow, etc. It also has hooks to talk to the Globus API as needed - just need a way to submit new tasks. These JSON files would serve the purpose of the tracking database mentioned above.
Remaining question is how we know when files in the gantry directory are ready to go to globus. @jdmaloney mentioned some length of time - say 15 minutes - when, if the file has existed for that long, we might assume the transfer is complete (i.e. from the USB hard drive coming off the field). Not sure if something like inotify could be more graceful in recognizing when that transfer is complete.

We need to know when the file transfer is finished. We can probably use IN_CLOSE_WRITE of inotify, but the problem is that inotify does not do recursive checks. I ran into this problem with crashplan where it needs to put an inotify on each folder and each file to know when a file is modified, or a new folder/file is added. The result is that the sytem ran out of inotify handles.
One other options is to use a simple find find /home/gantry/data -mmin +15 -type f -print to find all files modified more than 15 minutes ago. Check to see if file is already in you json file, if not then transfer, add the transfer info to your json file (and write to disk). Next check with 2 to see if any transfers are done, for those that are done, move the files to /home/gantry/deleteme.

Just to clarify, is (2) being implemented in #62?

yes

I can give an update this week, but I have some big chunks of this done. the code is in my branch here: https://github.com/terraref/computing-pipeline/tree/globus-monitor-api/scripts/globusmonitor
1 MAC transfer initiator
gantry_monitor_service.py, config_gantry.json

I have a service that will monitor a folder on disk for new files.
When new files are ready, this will add them to the pending queue. (haven't written logic checks for determining readiness yet - we can talk about how this should work)
Pending queue is used to start a globus task. Right now, all files in queue are sent in one task. I put a TODO to discuss how this should be handled, especially re: metadata. This will depend on how files from gantry are structured - my code supports metadata, but nothing about how we're going to batch things together.
Once (2) API reports transfer complete and Clowder upload done, move the files to a folder where they can be deleted.

2 NCSA Transfer Monitor API
setup.sh, globus_monitor_service.py, config.json
I patched up the last couple pieces of this, now supports full flow of monitoring directory -> sending Clowder notifications when files are done -> using local file upload with metadata to create Clowder entries and writing log files.
remaining

figuring out how we bundle things together and how we handle metadata. will metadata be at the file level or the dataset level?
will we be sending an entire directory of e.g. 12 files at a time, or more fragmented?
do we rely on metadata to determine creation of datasets/collections, or should this be part of gantry script in some other way? will this result in needing to check whether dataset of a given name exists, in case members of dataset are transferred across multiple globus jobs?

I'll answer based on the way the raw data are being written out, which was discussed in terraref/reference-data#2

figuring out how we bundle things together and how we handle metadata. will metadata be at the file level or the dataset level?

metadata will be provided for each 'dataset'. A 'dataset' is defined as a single time point captured by a single sensor (or group of related sensors in the case of, e.g. weather data and stereo cameras). see terraref/reference-data#2

will we be sending an entire directory of e.g. 12 files at a time, or more fragmented?

Lets start with the assumption that we send at least an entire directory at a time. Or, even if we send the directory in parts, that the script would work on the whole directory once it is available. I think the idea is that the lowest level directories (under sensorname/YYYY-MM-DD/YYMMDD_HHMMSS/) are atomic.

do we rely on metadata to determine creation of datasets/collections, or should this be part of gantry script in some other way?

Datasets will be defined as above (there is one metadata.json file per dataset).
Collections and other groupings will have many rules. Lets start by creating

one collection per sensor
one collection per day

Hopefully it will be easy to define and create collections on an as-needed basis.

will this result in needing to check whether dataset of a given name exists, in case members of dataset are transferred across multiple globus jobs?

Is there a reason to split up datasets?

That helps, @dlebauer.
I don't think there's a reason to split up datasets unless datasets start to get extremely large which seems unlikely. The remaining question would be how a dataset is recognized as ready to transfer on the gantry side - "once the whole directory is available". One file in a ds could exist for 15+ minutes before another file has completed transfer - we don't want to process the former until the latter is also there. Sounds like for now, we won't even _transfer _the former until the latter is also there.
Could be recognized when:

when some hard-coded set of files are present, e.g. "2 stereo bin files, 2 VIS files, an NIR file..." and we require all of them to be present. I don't like this one.
the metadata includes some sort of register that lists all the files included - we don't transfer the ds until we have its metadata and we have every file listed in the metadata
other ideas?

@max-zilla how about assume folder at time t is finished when when the folder at time t + 1 is created?

This would not work since that would leave the last folder of the day pending until the next day.

I think we should allow for files to be added to the dataset. We can find the dataset based on some metadata we store with the dataset. This way we can upload a file and add it to a dataset that was created earlier.

I would propose this general process:
/gantry/sensorname/YYYY-MM-DD/YYMMDD_HHMMSS/file.bin

once a file is ready in gantry, look at folder path to determine sensor and timestamp (YYMMDD_HHMMSS).   will be name of dataset.
queue file for transfer in globus. POST transfer details to NCSA monitor API, including dataset name the file should go into.
if we find a JSON file, assume that is metadata corresponding to dataset. don't send json file itself.
in case all we find is a json file without any new data files to go with it, i can add an endpoint to NCSA API that allows one to send metadata for a dataset without a corresponding globus task.
on NCSA side, when we get a finished globus task, loop through each file in the task. if file's dataset doesn't exist yet, create it. then add file to dataset.

This way we don't need to wait for a dataset to be complete before transferring, but we don't create a Clowder dataset twice if the component files get split across two Globus transfers.

That sounds good, but the met / environmental data is sent as .json files (terraref/reference-data#26). We could use a rule the file is called metadata.json (or perhaps metadata* is better)

Just created a pull request this afternoon with initial code. It'll still need some polish but I think it's close to ready.

@robkooper I want to test the services outside my local environment, but Globus Plus is required for transfer between two personal endpoints. You suggested using either Roger or Campus Cluster - I was able to go through the RSA process and get access to Roger's globus endpoint.
I want to be careful in the next step. I can SSH into Roger and git clone the computing-pipeline repo to get my script in my home dir, and create /gantry directory to act as the incoming gantry folder. Questions...

Do you have to use the batch job stuff on Roger no matter what? Is it possible to simply run my python script straight from the command line? Nothing needs to talk to the gantry script from the outside (only vice versa) so we shouldn't need to expose any ports or anything.
if that's possible, what about dependencies? The python requests library for instance. I also thought I could use docker to bundle these things, but docker is apparently not installed on Roger either. What's the proper procedure here?

Did you try qsub -I for interactive node?
@jdmaloney or @yanliu-chn thoughts?

@max-zilla use openstack to create a docker image. You should be able to use the existing coreos image we have for the toollauncher (running on nebula) or you can create a new coreos on roger and launch it there.
You can monitor the data going to roger, but you can not transfer it there using your code (unless you get the secrets from @jdmaloney for the transfer user). You can however do the transfer to campus cluster and monitor it to see if everything works as expected.

@max-zilla I can get you set up to transfer to Roger this afternoon.  I think it would be good to do transfer testing on there directly as that's where we're going to be putting the data.
Speaking of data, I've written a temporary parallel transfer tool that is moving data between the gantry and here for now (we're up to 5.4TB of data, and now accumulating multiple TB a day, the gantry seems to be outputting at or close to full capacity).
I have a meeting here at 1PM, but I can try and find you after that Max

@jdmaloney I was actually planning on transferring from Roger -> my laptop endpoint as the test so I wouldn't need to write to roger right away, but would be good to set that up either way.

@robkooper @dlebauer I just stopped by and talked to @max-zilla I found a handy bit of information this morning.  The program linux uses for setting up the ftp server (vsftpd) has the built in ability to log incoming and outgoing transfers.  I've had this logging on since Day 1, and it prints the full path for the transferred file in the log as well as the completion status of the transfer (either c for complete, or i for incomplete).  Essentially this will allow us to scrape that log file for all lines ending in "c" and then grabbing the file path off that line to add the "to-be-transferred" list.  Since there are no outgoing transfers via ftp, this log file will hold exactly what we need to move.  It is located in /var/log/xferlog

And the logfile that @jdmaloney describes should mean we don't need to ask LemnaTec to modify their transfer script - the FTP action will appear in the logfile regardless and we can just initiate transfers based on that.
Only question would be, how big will that logfile get, when are things purged from it, best way to avoid duplication of transfer. Do we think about e.g. every week moving the log file to some archive and beginning a new one, or is that already solved for us by vsftpd?

@max-zilla I can set it up with logrotate.d so that every day a new log file is created and the old one is either moved over (newest log file is just xferlog, all others are xferlog-yyyymmdd), or both moved over and compressed to save space.  Both are very easy, just a couple lines in a conf file.  This way your script only has to consider a day at a time, would that be beneficial?
In terms of size, all transfers up to this point encompass ~1.3 million files (lines in log file), which equates to about 250MB of log file.

@jdmaloney do you control the ncsa#roger-gantrytest globus endpoint? I'm running into trouble initiating transfers in python from ncsa#roger - I get a 409 Conflict expired credentials error, as far as I can tell it's because Roger has this as ID provider:
Identity Provider
Type   MyProxy OAuth
Hostoa4mp.ncsa.illinois.edu

...while the globus test endpoint I've successfully used before in a similar way is:
Identity Provider
Type   MyProxy
Hostmyproxy.globusonline.org

Their Oauth documentation talks about web applications routing through a login page to get an oauth token, it isn't discussed on the API side and I don't want to dig into that yet if I don't need to. Thought maybe the gantry endpoint might have different settings (although maybe not, it lists ncsa#roger as Host Endpoint).
Could you grant me access to ncsa#roger-gantrytest if possible, so I could see if it works any differently? Thanks.

@robkooper @max-zilla Just wanted to update the thread.  Max and I talked about the above comment.
Also I just finished analyzing the logs from the gantry over the past couple days.  Lemnatec is doing a fairly continuous dump of the EnvironmentLogger data, however, all the moving sensor data (the vast majority of data) kicks off around midnight and continues until it is finished dumping that day's data (takes 4-9 hours so far depending on the day) and then stops and waits until the next day.
Thought this might be helpful as we try to figure out how often/when/how to grab data off the server.  If Lemnatec is only doing the sensor dump once a day, it may not make sense for us to run continuously.

@markus-radermacher-lemnatec at what frequency are you planning to dump sensor data?

@jdmaloney where is the log file?
Do you or @markus-radermacher-lemnatec know what sensor(s) are producing the large number of files?

@dlebauer Log file is at /var/log/xferlog on the gantry cache server; below is a snapshot of the sensor data quantities at this time.  This is an accumulation of all data gathered so far.

@dlebauer Misread your question, file counts are now below as well.  File count is actually just shy of 1.1 million

email update from 04/12:
JD and I had discussed Thursday as a possible point to pause the manual transfer from the gantry and try to slide in my monitor code. To that end, I’m trying to test the heck out of it with as many edge cases as possible, but generally the sooner the better to start getting things working at the other end while the data is less critical if we lose a file here and there.
Log file scraping
This morning, I continued testing the log file scraping and I think we’re in pretty good shape. The basic logic is:

     Get most recently read log line from monitor_status.json file (initially will be blank on day 1)

o   Get FTP transfer timestamp from that line (“last read”)

     Check for “xferlog” file in the configured folder and read the first line

o   Compare [first line] timestamp to [last read] log line
§  If [first line] is before or equal to [last read] time, our resume point is somewhere in this file.
§  If [first line] is AFTER [last read] time, there’s a gap in our logs and we need to walk backward to an older log file and get the entries we might’ve missed.
o   If our resume point is in this file, iterate through the lines until we find the line that matches our most recently read log line. Then starting on the next line, grab the rest of the entries that end in ‘c’ (completed) and queue them for transfer.
o   If we need to walk backward, we expect the following naming structure:
·         xferlog                  (latest/current log file)
·         xferlog-1
·         xferlog-2
·         …
·         xferlog-4.gz        (eventually these get g-zipped, not necessarily the 4th one back but somewhere)
…so recursively check each file the same way (look at first line, is it here? If so, scan for the last line and start from there). However far back we go, we scan the rest of the backlog file(s) and then the current one. So if my last scanned line is found in the xferlog-2 archive file above, it’ll scan all the lines from xferlog-2 after that one, then all the lines from xferlog-1, then all the lines from xferlog until we either reach the end of the file or find an incomplete transfer entry. It’ll auto-unzip the .gz file if necessary.
EDGE CASE: If we have a last-read log file but we go allllllll the way back through the logs without finding it (this should never happen), it’ll just start from line 1 of xferlog.
The way I’ve tried to write it, we don’t really care where in the log files our record is as long as the files follow this naming convention – we’ll look as far back as we can to find the proper resume point, then get everything we can that’s ready.
Globus size management
Gantry monitor now has a config entry max_transfer_file_count that defaults to 100 – if we have more than that many files queued for Globus transfers, we’ll batch them into that size. So 200 files with a 100 max size would be chunked into 2 transfers. Don’t know what this magic number will be, but intended to balance good utilization of Globus with not making transfers too big so Clowder has to wait for 5000 files to transfer via Globus before it can start ingesting.
Crash testing
Trying to think of all the ways this could fail and handle gracefully. Examples include:

     Gantry can’t talk to globus

     Gantry can’t talk to NCSA

     NCSA can’t talk to globus

     NCSA can’t talk to Clowder

     404/500/other responses from Clowder or Globus or other

     System crashes during action <x>

Part of this is being addressed via the .json log files, which periodically write data objects to disk (pending transfers, active transfers, created datasets, last read FTP log line, etc). The monitors have hooks on initialization that will look for these files and load the data it finds to resume from a previous point. These also keep a .backup copy when the file is updated. That said, I’m positive there are gotchas we’ll run into and I wanna make this as bulletproof as possible.
Clowder data structure
I added some basic methods to create clowder datasets/collections as we go, akin to my script for uploading the local Roger files. You can define a single space in the config file where everything will live (but it can be blank). There will be one collection for each Sensor, one for each Date, and that’s all I’ve added so far. If we want to create collections by metadata etc. I have some things in place to support that but haven’t started down that path yet.
Purging gantry cache
My config file right now just allows you to define a folder (default /home/gantry_delete) where gantry files will be moved once Clowder confirms the whole pipeline was completed for that file. This has not been tested at all yet, and I created the folder for now rather than outright deleting files for safety reasons. We can discuss what should happen here.

Todos that remain which may not be done before first rollout:

Collate the Environmental Logger JSON files. There are a ton of these and they're small - either LemnaTec or us should combine these into one longer file before transmission.
Refinement of Clowder organization - are there additional collections/spaces that should be created from metadata? Rob and I discussed a 'smart collection' notion that would be potentially defined by some XML structure to say "create collection with this metadata field as first level, then this metadata field as second level, etc" to automatically create a nice tree structure for navigating.

Will contact Globus to ask about:

Unexpected 409 NoCred ClientError (credentials not authorized) when initializing Globus transfer via python API. Encounter this error sometimes even after refreshing goauth credentials - error stops if I go to Globus site and go to Transfers page & select endpoints there. Don't even need to do a transfer from browser, just going to that interface does the trick. I wonder if there's an undocumented handshake of some kind I can do with API to avoid this.
Is it possible to sync timestamps between source and destination? In browser UI there is a checkbox to do this in a transfer, but I haven't found a way to do it via python API.

Proposed changes to the environmental logger file are listed in issue reference-data#26

Answers regarding Globus from @bd4 (via email):

The 409 NoCred error means that you need to re-activate the source
and/or destination endpoints, likely because the credentials they were
activated with have expired. It's a confusing name, but it's
referring to the endpoint activation credentials, not the credentials
used to authenticate to the REST API.

When you activate an endpoint, it saves the credential with the
endpoint, for the entire user identity. This means that if you use an
endpoint via www.globus.org, you will end up activating it, and that
will affect any Transfer API scripts that are running under the same
identity. Using the website concurrently with testing your script is
one possible reason for the 409 NoCred going away.

For preserve mod time, the option is called "preserve_timestamp". It's
listed in the docs here:
https://docs.globus.org/api/transfer/task_submit/#transfer_specific_fields

I'm going to look into using
api.endpoint_autoactivate(<endpoint_id>)
when this error is encountered to hopefully avoid the endless error state I was running into before.

Autoactivate will help, in that it will use credentials available for other endpoints if available. For example, if you have used one XSEDE endpoint on the web and activated it, any other XSEDE endpoint can be autoactivated, by copying the credential to the second endpoints activation record. This is basically a hack to make activation look more like it's per identity provider, when the model we chose initial was per endpoint.
The thing is, that one credential that is used to activate the first XSEDE endpoint (when you get redirected to the XSEDE portal and have to sign in) will eventually expire. At that point, a new credential will need to be acquired somehow. You can't escape activation completely, unless your application is limited to using shared endpoints and Globus Connect Personal endpoints, which can always be autoactivated because they essentially rely on Globus as their identity provider (and don't require authenticating with a third party IdP).

@robkooper @dlebauer
After a few last-minute things that needed to happen, the plan is for JD and I to switch over to my gantry monitor service on Monday morning.
@jdmaloney  – you installed docker already, so on the cache server here's what we need to do.
Copy over a prepared version of config_custom.json with some key parameters filled in - I'll leave a 2nd comment to address this. The folder where you put this config file will also be where the /log and /completed files are written - this will be LOG_DIR below.
On the cache server:
docker pull maxzilla2/terra-gantry-monitor
...to pull the image from dockerhub.
–v LOG_DIR:/home/gantry/data
-v /gantry_data/LemnaTec/MovingSensor:/home/gantry/sensors
-v /gantry_data/deleteme:/home/gantry/delete/sensors
-v /var/log:/var/log
maxzilla2/terra-gantry-monitor```

...This will run the container. The -p flag will expose that port on the Gantry cache - **this has no authentication barrier** because it's intended for internal use only by the Gantry folks. In practice if the FTP log scraping is behaving we probably won't ever need to do so, but this allows one to POST file(s) to be transferred to globus at ```localhost:5455/files```

The -v flags will mount 4 directories on the gantry cache inside the docker container:
1. directory where all the monitor log stuff and config_custom.json go.
2. directory that contains MovingSensors and the other raw data
3. directory where files will be moved (in same subdirectory structure) to be deleted
4. directory where xferlog, xferlog-1... xferlog-N.gzip are written from FTP.
The paths after the : are the default config values, so if we don't overwrite them in config_custom that's where the script will expect to find things inside the docker container. We just need to mount external directories inside the docker container at the correct paths.

We start up the NCSA piece on Roger with the proper credentials. I can test a good chunk of this beforehand. At that point we need to confirm:
1) The gantry script can ping Globus, Roger and our API successfully
2) The Roger script can ping Globus and receive messages from the gantry via the API
3) Globus endpoints are running

...then, we hold our breath and send a file via FTP to start it up.

GANTRY SIDE
The script will load parameters from config_default.json, then update any overrides from config_custom.json. The default values I think we can avoid changing:
{
  "log_path": "/home/gantry/data/log/log.txt",
  "active_tasks_path": "/home/gantry/data/log/active_tasks.json",
  "pending_transfers_path": "/home/gantry/data/log/pending_transfers.json",
  "completed_tasks_path": "/home/gantry/data/completed",
  "status_log_path": "/home/gantry/data/log/monitor_status.json",

  "gantry": {
    "incoming_files_path": "/home/gantry/sensors",
    "deletion_queue": "/home/gantry/delete/sensors",
    "ftp_log_path": "/var/log",
    "file_check_frequency_secs": 120, ----- how often the FTP log is read for new files
    "globus_transfer_frequency_secs": 180, ----- how often to bundle Globus transfers
  },
  "globus": {
    "authentication_refresh_frequency_secs": 43200, ----- refresh Globus auth
    "max_transfer_file_count": 100 ----- max number of files for 1 Globus transfer
  },
  "ncsa_api": {
    "api_check_frequency_secs": 180 ----- how often to ask Roger API if a transfer is done
  },
  "api": { ----- this is the little API to manually submit jobs to gantry monitor
    "port": "5455",
    "ip_address": "<IP_ADDRESS>"
  }
}

...and the ones we might want to override:
{
  "globus": {
    "destination_path": "/home/projects/arpae/terraref/.../", ----- where to put Globus transfers
    "source_endpoint_id": "...", ----- MAC endpoint
    "destination_endpoint_id": "...", ----- ROGER endpoint
    "username": "",
    "password": "" ----- credentials of Globus account that will do the transfers
  },
  "ncsa_api": {
    "host": "http://<IP_ADDRESS>:5454" ----- Roger API (the other half of this puzzle)
  }
}

ROGER SIDE
Same deal, the few default params that work:
{
  "globus": {
    "authentication_refresh_frequency_secs": 43200,
    "transfer_update_frequency_secs": 120 ----- how often to ask Globus if transfers done
  },
}

...and ones to change:
{ ----- basically we replace ROGER with /path/to/folder/for/logs
  "log_path": "ROGER/data/log/log.txt",
  "active_tasks_path": "ROGER/data/log/active_tasks.json",
  "completed_tasks_path": "ROGER/data/completed",
  "dataset_map_path": "ROGER/data/log/datasets.json",
  "collection_map_path": "ROGER/data/log/collections.json",
  "status_log_path": "/ROGER/data/log/monitor_status.json",
  "globus": {
    "incoming_files_path": "/home/globus", ----- needs to match above - where to look for Globus files
    "valid_users": { ----- list of whitelisted Globus accounts that can POST to this monitor
        "GLOBUS_USERNAME": {
            "password": "",
            "endpoint_id": "..."
        },
        "GLOBUS_USERNAME2": {
            "password": "",
            "endpoint_id": "..."
        },
    }
  },
  "api": { ----- IP and port for this service on Roger
    "port": "5454",
    "ip_address": "<IP_ADDRESS>"
  },
  "clowder": {
    "host": "http://<IP_ADDRESS>:9000/clowder",
    "user_map": { ----- this maps Globus account -> Clowder account that will be owner of files
        "maxzilla": {
            "clowder_user"<EMAIL_ADDRESS>            "clowder_pass": ""
          },
            "GLOBUS_USERNAME2": {
            "clowder_user": "",
            "clowder_pass": ""
          },
    },
    "primary_space": "..." ----- UUID of a space all things should be added to, if any
  }
}

Monitors are currently running, but there won't be much activity over the next couple days while they plant in the field it sounds like.  We're going to try a couple FTP tests in the meantime.
Once the basics are confirmed functioning, there are still refinements to be made. One is to make sure the FTP logs are being archived/handled as expected. right now code expects "xferlog", "xferlog-1", "xferlog-2"... for progressively older FTP logs, but the original setting was writing them as timestamps like "xferlog", "xferlog-20160410", "xferlog-20160418", etc. either I would need to adjust my script to intelligently look back across the dates, or the FTP settings need to be adjusted to use the -1, -2 naming convention.
Will create issues for ToDos once the basic mechanism is tested some more.

I wrote up the deployment process in a google doc for now:
https://docs.google.com/document/d/1TxBOgaJBYd5YA-c0J2B11XU4Vm_jYc1PPgpeSLmAQD4/edit?usp=sharing

And the pull request for the new monitor code (in the computing-pipeline repo) is here: https://github.com/terraref/computing-pipeline/pull/83

Pipeline has been up and running for about 16 hours now. It's still processing the substantial amount of backlog on Roger so it hasn't yet tried to scan a metadata file - not sure yet if @jdmaloney's permission adjustments will fix that issue. I bumped up max Globus transfer size to 500 files per, and put restrictions on max # of active transfers and pending files at a time to avoid running out of memory on cache like what happened on Wednesday.
I paused the bulk upload script that was processing the older files JD manually moved from cache in April - this is just a dumb spider that crawls all directories as we hadn't had any new data yet, but now that new data is coming in I'll need to refactor to only get the older files and not start re-uploading the files arriving via gantry monitor.
6.8 TB of data so far.

@jdmaloney one thing I noticed is that Globus autoactivation in code wasn't working - I was still needing to authenticate the gantry endpoint in the Globus UI periodically. I looked into their source code and realized they will not store the gantry creds for autoactivation - this would be the data_mover creds, not my maxzilla globus account.
To use manual activation I need 3 pieces: user/pass for data_mover (OK), and a 'proxy_chain' cert that is very poorly documented but as I understand it, they have the public key piece already:
{
		'name': 'public_key',
		'DATA_TYPE': 'activation_requirement',
		'required': False,
		'value': '-----BEGINPUBLICKEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA5VQX1Rr26MFo7cyYAInr\n+e5w3tMDz9H6R1T5DFGi0PKTKB7OVPdzHnzeP/Bc00BKKNcJrjM57h1PnjkbsY76\n3GEPlOJUuWCTbzrYhMjreD1pp1BWy7W+pdfjtvNumg2SVTxS0axDUVDaV8mWtqW4\nMVtmKoCjTKxrhTUP3VZ5q3iA+gzxxxpqX5yrjfSVk+QsPhkPQ8yBAJ32CWJI1riu\nzz1FvJw7UMqLa8QrSmA01jCJlsEjLxVfrcoYFB1uofBNg0tr7VNfWe7n7Ed0Iksb\n3faxgNIgr1ta7zQwYd9vhjJh87H0UQmv3Vn9pK3ZCLrqjt8mD8mWLY1HK3z1Mi8L\nLHTvI4xAwmClmi4qxbWH5iKPezP2zMFeSXKFWt7xz6ZIxYSEt8j0CCb6vJLCm3Wu\nFd9HiMeXA341ufaK/8m4FlA2aBPuiIN2ULlYC9BvAzmOhkieUgYwET+53IC89vi+\n60CFd1s1CeRtKSJ3hGso+ztErv6rILBZfZ0Aq2Da8ptqq0JjbmmOW3KYdC+cl3uR\n9VjFRMHhytp3SVQL9RrAv4Gv8pcu1JHeLp/Y7FG0sr//hISbs/o8Yj0mwoYRfE/r\nXc9ntErbytIbJGG3wAHQ2dt/+EqpF/v0yDRIUEub4mOrBZKgXVojH8i2uGt/Y8S6\nG8CZ11hEpf8zHxVKl1rCk3kCAwEAAQ==\n-----ENDPUBLICKEY-----\n',
		'private': False,
		'ui_name': 'ServerPublicKey',
		'type': 'delegate_proxy',
		'description': 'ThepublickeyoftheGOAPIservertouseintheproxycertificatefordelegationtoGO,
		inPEMformat.'
	},
	{
		'name': 'proxy_chain',
		'DATA_TYPE': 'activation_requirement',
		'required': True,
		'value': None,
		'private': False,
		'ui_name': 'ProxyChain',
		'type': 'delegate_proxy',
		'description': 'Aproxycertificateusingtheprovidedpublickey,
		inPEMformat.'
	},

See the last description - "a proxy certificate using the provided public key, in PEM format". I am assuming the public key above was created for data_mover on the gantry, as I did not create/provide it - it was returned by Globus.
So I'm trying to figure out which cert to use, none of the obvious ones I have locally are working but this isn't really my strong suit. I don't quite understand if this proxy cert already exists somewhere, or if it needs to be created using the private key (https://github.com/globusonline/transfer-api-client-python/blob/master/globusonline/transfer/api_client/x509_proxy/m2.py). I can reach out to Globus but thought I'd see if you know where this public key came from.

@max-zilla The cert does need to be created using the private key. Proxy certificates are certificates 'issued' by a normal end entity cert, instead of a by CA, but the process is similar to a CA issuing a certificate. In this case creates a new certificate using the public key provide by the Globus Transfer API, issued by the credential you provide, and signed by the corresponding private key you provide. It will also only work if the end entity cert you create the proxy from is trusted by the endpoint - the resulting proxy certificate is a proxy for the original certificate, and has the same authentication power. Sorry if that's confusing, there is a lot of X.509 background here and I'm not sure what your familiarity is.

Thanks for the quick response, @bd4! I think I understand what's necessary. My familiarity is fairly nonexistent but it's improving, I'll follow up if I have more questions.

@max-zilla what is the status of this issue? what tasks remain before this can be closed?

Yes, we can close this. The pipeline is functional - I will create a separate issue for the remaining adjustment that might be needed (related to Globus authentication certificates) but not sure if this is necessary yet. Will close shortly.

https://github.com/terraref/computing-pipeline/issues/105 new issue here. Closing this one.

@max-zilla have the / will the relevant information from this thread be captured in the documenation?

@dlebauer yes, I have a very rough draft (mostly copied from here) of a pipeline document in my fork of the terra-ref documentation. Once I make it more presentable we can merge into main docs.


Add Kube from SpaceDock

This pull request was automatically generated by SpaceDock on behalf of Katten, to add Kube to CKAN.
Mod details:
name = /mod/1539/Kube
author = Katten
abstract = This mod adds parts meant for larger space-ship kind of vessels, built in a modular free form way.
license = MIT
Homepage =
description = Gosh, going to space is hard! And yet, with all the research going into space traveling, we are still stuck with those old fashioned round tanks and engines!
Well, thanks to the superb engineers here at m3tech, we are now proud to present a true breakthrough in space travel - the Kube!
It consists of six sides, all of which are the exact same size - enabled by our long time expertise in advanced modelling and nano materials!
But wait, there is more! See those edges on that cube? They are ALSO the same length - all 12 of them! Fascinating stuff indeed.
We also supply triangle shaped and rounded pieces, with many converters between the different formats. Some of them are also available in truss format!
For all your space storage needs, we got you covered with our lineup of cargobays, on all your favourite shapes.
Here are most of our available parts:

These products lack all forms of safety certifications, but have totally been tested in space, as you can see here:

Please direct questions about this pull request to Katten.

Replaced with LGG versoin


discord-ptb: 0.0.38 -> 0.0.39

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for discord-ptb is: All-in-one cross-platform voice and text chat for gamers
meta.homepage for discord-ptb is: https://discordapp.com/
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.0.39 with grep in /nix/store/c5klcjr96c8x2i7ap71s7rbryw525bsk-discord-ptb-0.0.39
found 0.0.39 in filename of file in /nix/store/c5klcjr96c8x2i7ap71s7rbryw525bsk-discord-ptb-0.0.39

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
discord-ptb

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/c5klcjr96c8x2i7ap71s7rbryw525bsk-discord-ptb-0.0.39 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A discord-ptb https://github.com/r-ryantm/nixpkgs/archive/7988ae64f5a5221b20c10ffcbebe96190fa5ee23.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/c5klcjr96c8x2i7ap71s7rbryw525bsk-discord-ptb-0.0.39
ls -la /nix/store/c5klcjr96c8x2i7ap71s7rbryw525bsk-discord-ptb-0.0.39/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

discord-ptb

Maintainer pings
cc @MP2E @devins2518 @Artturin @Infinidoge for testing.

Please reopen.


couldn't parse packet from <IP_ADDRESS>:59464: packet has non-zero reserved bits

Using spotifyd  but I believe these messages originate in librespot, I'm getting quite a few of the above message, I suppose these are other clients in the network, anyway to quiet these down or some way I can fix whatever is wrong? Thanks.

Spotifyd uses an old version of librespot they would need to update from 0.2 to 3.1 to even be able to tell if this is still a valid issue.

Can you post the full log? I think this is from libmdns?
Also is this just a nuisance or is some behavior actually broken?

It's absolutely from libmdns:
https://github.com/librespot-org/libmdns/blob/acd405604d2fb589e9848d1292ce9404bd206d91/src/dns_parser/error.rs#L17

Spotifyd uses an old version of librespot they would need to update from 0.2 to 3.1 to even be able to tell if this is still a valid issue.

Sorry I didn't check the versions before hand, I saw a similar issue and it was redirected to librespot so I just came straight here.

Can you post the full log? I think this is from libmdns?
Also is this just a nuisance or is some behavior actually broken?

That's pretty much all I see, just repeated many times. Not sure if it's an actual problem and if this may be related but I was often loosing connection through spotifyd necessitating a restart of the daemon, that's what led me to check the logs and this was pretty much all there was.

Can you post the full log? I think this is from libmdns?
Also is this just a nuisance or is some behavior actually broken?

That's pretty much all I see, just repeated many times.

Again please post the full log. Just copy paste everything.

Not sure if it's an actual problem and if this may be related but I was often loosing connection through spotifyd necessitating a restart of the daemon, that's what led me to check the logs and this was pretty much all there was.

Please try stock librespot instead of spotifyd and see if you have the same issue. I think you will, but just to rule that out.
Then when we certify that this issue is with libmdns you should report it here: https://github.com/librespot-org/libmdns

it’s a beacon in my home mesh network.
On Mon, Dec 20, 2021 at 9:34 AM Jason Gray @.***> wrote:

What device is <IP_ADDRESS>?
—
Reply to this email directly, view it on GitHub
https://github.com/librespot-org/librespot/issues/911#issuecomment-997978542,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC7TMS7BGUR4CUENDIICKKDUR45HBANCNFSM5KJJNLYA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

What device is <IP_ADDRESS>?

It's a beacon in my home mesh network.

Can you post the full log? I think this is from libmdns?
Also is this just a nuisance or is some behavior actually broken?

That's pretty much all I see, just repeated many times.

Again please post the full log. Just copy paste everything.

I no longer have it but I'll reproduce it later today.

Not sure if it's an actual problem and if this may be related but I was often loosing connection through spotifyd necessitating a restart of the daemon, that's what led me to check the logs and this was pretty much all there was.

Please try stock librespot instead of spotifyd and see if you have the same issue. I think you will, but just to rule that out.
Then when we certify that this issue is with libmdns you should report it here: https://github.com/librespot-org/libmdns

Ok, I'll need some more time to figure out how to do this, will reply when I've done it.
Thanks.

What device is <IP_ADDRESS>?

It's a beacon in my home mesh network.

Well either the beacon is misbehaving or very possibly libmdns is misbehaving or just being overly chatty in it's logs?
If it's not actually causing any trouble other that spamming the logs I would just drop the log level to warn or error.
Otherwise I would ask the Spotifyd team to help fix libmdns, it could use some love for sure, or alternatively maybe since Spotifyd's stated goal is to be "An open source Spotify client running as a UNIX daemon." maybe they should use a different mdns implementation since they don't care about Windows compatibility?

Well either the beacon is misbehaving or very possibly libmdns is misbehaving or just being overly chatty in it's logs?

If it is then avahi and systemd-resolved don't seem to be bothered by it, I use both in different pc's.
I actually stopped using spotify-tui/spotifyd due to the connection drops which may not be related to this but this was the only thing I could see in the logs, I also tried Spot that also uses librespot and had a similar problem and they are on 0.3.1 and don't think they have the same network spotify receiver feature that spotifyd/spotify-tui has. Please feel free to close this issue if you think it's a red herring, or for sure belongs in libmdns (I think it does).

If it is then avahi and systemd-resolved don't seem to be bothered by it, I use both in different pc's.

libmdns is mostly functional but is a little rough around the edges. There have been other issues and talk of improving it, but in my mind the best option is to just hook into the underlying systems implementation (avahi in the case of Linux) and not have our own full mdns implementation. That was also in the works but stalled.

Please report this to the separate libmdns repo. In think it was @willstott101 who did excellent work on that but is not active anymore. I consider improving it as a nice challenge, but like @JasonLG1979 says, it’s mostly functioning now and I’ll be spending my time on many other librespot features first.


Add Kristina Devochko as Tech Lead

This PR updates the README and Website to include @guidemetothemoon after the nomination closed with all TOC liasions gave a +1; see https://lists.cncf.io/g/cncf-toc/topic/nomination_of_kristina/102766241
fyi @mkorbi @catblade

@catblade thank you! I'm excited for contributing even more in this new role😻
Thank you for following up this whole process @leonardpahlke !


UserDetailView fails on mixed-case usernames

UserDetailView fails to find the user based on slug if the username is not stored in lowercase, which is likely to be the case given that django-allauth preserves case when storing usernames by default.

I encountered something like this not too long ago. Let's see if we can write a test for it, shall we?

We couldn't reproduce in #2234 so I'm closing this.


Surrogate pairs in JSON strings

The JSON parser doesn't notice if a string contains surrogate pairs, either as a pair of \u escapes or as a pair of UTF-8 encoded versions of the two halves.

Will be fixed in https://github.com/mapbox/tippecanoe/pull/486


Use packaged chromium source to speed up CI builds

This PR adds the capability of using a tarball cache of the chromium source code to speed up builds.
The tarballs are stored on S3 by platform and chromium version (eg libchromiumcontent/linux/src/63.0.3239.150/src.tar.bz2).  If the tarball doesn't exist for the version/platform, then the source will be downloaded using gclient sync.  In this case if the upload_packaged_src argument is passed to script/update, then the source will be archived and uploaded to S3.
To use this feature, pass the use_packaged_src argument when calling script/update.
The chromium source that is archived does not have the .git directories or the native_client directory.
The tarball sizes for 63.0.3239.150 are as follows:

Linux - 1.9 gb
Windows 1.5gb
OSX - 1.4gb

It may be possible to further optimize these tarballs for size, but I wasn't sure what other files from chromium source we can drop.

Just for your information, we were doing this before and I had a service providing universal source tarballs https://github.com/zcbenz/chromium-source-tarball.
I dont remember the reason and when we switched away from it (I was not making the call), but if you are interested in it, I can bring my service back to life.

@zcbenz would your solution have any advantage over this one?  It seems to me that using S3 means we don’t need to run another service.

It hosts assets on GitHub Releases so it only needs a $5 cloud machine to run. It does not make much difference for us since the bill is on GitHub anyway.


sqlcipher_flutter_libs does not compile

I'm trying to compile sqlite3.dart package with Flutter 2.0.3 installed.
I have errors in sqlcipher_flutter_libs.dart

If I add Flutter dependency in the pubspec.yaml, it is working

Is it normal ?

Sorry, I should have caught that myself. This is fixed now, thanks for the report.


Localization not working on self-contained single-file.

The reason is the resx files are embedded on the executable file and thus there is no satellite assemblies for each culture in the folder.
The solution is to return a list of cultures that we know are supported. However, if the project contains the <InvariantGlobalization>true</InvariantGlobalization> setting only the invariant culture is supported. For other cultures to be supported, it is necessary not to use this configuration or use it as <InvariantGlobalization>false</InvariantGlobalization>.

You mean that a consuming application that publishes as single-file is having that trouble, right?

You mean that a consuming application that publishes as single-file is having that trouble, right?

Yes.

You mean that a consuming application that publishes as single-file is having that trouble, right?

There was also a bug for apps not publishing as single-file where the AppContext.BaseDirectory was not executing as it should. Previously, assembly.Location was used but a warning always appeared that it was not supported in trimmed. So I now chose to use assembly.GetName ().Name. Previously, this bug was not detected because SupportedCultures was only asserted with the GetSupportedCultures method, which always returns the same value regardless of whether it was incorrect. Therefore, I added the Assert.Equal (4, Application.SupportedCultures.Count) instruction to the unit test, whose value must be manually updated whenever a new resource file is added. I would prefer it to be automated but it is not possible. Only in applications that are published as non-single-file can it be automated because it is searched by the satellite assembly directories.

I would suggest actually doing a full AoT, not just single-file, build for this csproj. The reason is that AoT brings in even tighter restrictions which would be a good idea for us to comply with.
Thankfully, it's better with .net8 than it was even with .net7.
For debug symbols, you'll want to add, in a PropertyGroup: <StripSymbols>false</StripSymbols>, too.
For some reference material, while you're working on that:

This doc outlines a few known incompatibilities and some workarounds, in the single-file case.
And here are the AoT rules, which are on top of the single-file rules

In fact, thinking about it...
We should probably add another unit test project, at some point, specifically for testing AoT, to protect against us doing anything in the future that breaks people's AoT builds.

I already tried to add an unit test project but we only can create a Process but isn't possible to make the test on what matter.

Yeah, that'll likely require a bit more work. It can be addressed later, anyway, especially since single-file has to be working perfectly before AoT is even possible.

Also, you may find this document useful for this issue: https://github.com/dotnet/runtime/blob/main/docs/tools/illink/data-formats.md#descriptor-format

We also very likely will need to use a lot of [DynamicallyAccessedMembers] and related attributes on any non-sealed types, especially the various View types, so that derived types in AoT builds don't break and other related possibilities.
Alternatively, the descriptor and substitution XML files are likely to be less work and more durable, but just need a bit more careful design.

Another thought:
Rather than having a separate SingleFile csproj, how about just adding a build configuration to the sample projects that does single-file and AoT? Then you'll get confirmation that things done in them work, for free.

Also note that bool is not blittable, which has consequences for PInvoke and AoT.
Any types that contain bool fields are not blittable and may need appropriate marshalling attributes applied to ensure correct and consistent behavior for all build scenarios.

Another thought:
Rather than having a separate SingleFile csproj, how about just adding a build configuration to the sample projects that does single-file and AoT? Then you'll get confirmation that things done in them work, for free.

The others projects are examples for simple implementation. The SelfContained project is for test and ensuring that AoT and single-file are working properly. So, @tig is right when he said that examples must be the simplest possible. The SelfContained is really for testing stuff.

That's not relevant to what I'm suggesting.
The samples themselves don't need to change. What I'm suggesting is adding another build configuration, so that you don't have to do the work to verify that all of the things that the samples already cover again, in another project. And you'd need to do exactly that, otherwise, because simply being able to compile and run a barebones app does not prove that the library is compatible. It just means that the code covered in that app doesn't fail as written.
The nature of self-contained is a bit tricky, because it's still subject to JIT compilation, by the same rules as any other application, but unreferenced code that the trimmer removed won't be noticed until you call it, because no attempt to compile it to a binary has happened yet. Roslyn compilation generated a .net assembly, the IL linker walked the IL code graph and aggressively removed anything it couldn't guarantee was in use, and then it got packaged into a zip archive (single-file is still separate files, just zipped together and extracted at runtime, which is why they start up slower - especially on first launch).
But use of the existing unit test project isn't sufficient, because it touches so much stuff all at once that not much of TG itself is going to get trimmed. Only stuff where coverage metrics show 0% might get trimmed.
Although, a thought that just occurred to me is that we might be able to use build configurations on that to also target the trim scenarios. Here's what I'm thinking right now for that:
Basically, we would use conditional inclusion/exclusion of portions of the test project code, for each of several new build configurations. This would ensure we're allowing a different portion of the TG code to become trimmable for each different configuration, by not being referenced.
For running the tests, all you have to do is specify the configuration, which is also trivial on the github runners, using a matrix, and would happen in parallel.
Not directly associated with the above, we can make this all a lot easier on ourselves by moving some namespaces around, since you can control trimming at the namespace level as well. For example, the root namespace's members as well as various base types like ConsoleDriver, View, and configuration-related stuff would be the majority of what we need to explicitly mark as not trimmable, leaving the rest up to the consumer's application code to dictate at compile-time.
Rooting or otherwise exempting the entire assembly isn't a viable solution, as that's basically just ignoring the problem and removing one of the value propositions of trimming in the first place, which is size reduction on disk and in memory of the resulting assembly, whether it be single-file, R2R, AoT, or just explicitly trimmed without specifically using any of those features. Common examples of that last one are containerized applications or small embedded devices (think Raspberry Pi for example), both of which typically aim to be or need to be as lean as possible).

What that all sums up to is that claiming trim compatibility really can't be done without one or more of the following:

Sacrificing benefits of trimming.
Making code available to call and use which can't actually be called and used consistently between untrimmed, trimmed, single-file, R2R, and AoT, but which compiles just fine potentially in every one of them - maybe even without warnings.[^1]
Closely related to the previous point, creating incompatibilities that will only show up at run-time and when specific code is called.

These can end up being internal, external, or both.
These are highly likely to differ significantly, in some cases, and very subtly, in other cases, depending on how trimming is being used

Making it hard to provide a guarantee that things will work consistently or properly on future .net versions, with or without recompiling TG, the consuming application, or both.
Requiring explicit cross-compilation against OS- and architcture-specific Target Framework Monikers

That's not a HUGE deal specifically for just consuming TG in simple scenarios, since nuget handles that all automatically.
It is a big deal for many other reasons, though, such as:

Consuming applications intended for single package muti-platform deployment have to be much larger due to having to include multiple assemblies
We take on a non-trivial maintenance burden, since we'd need to ensure not only that cross-compilation works, but that consumption of the cross-compiled assemblies on the target platforms works as intended. That all quickly explodes into hundreds or even thousands of runs, for the CI workflows, which would exceed the free usage credits pretty quickly.

Unless one or more of the above concessions (or some others, since that's certainly not an exhaustive list) are made and very loudly documented, the goal of making TG trim-compatible or anything that depends on trim-compatibility isn't attainable to a degree that we should advertise/list/claim that TG is, in fact, trim-compatible, as a general statement.
But if we do some (hopefully not terribly onerous) work to provide at least some hard proof to support more specific claims, such as any of the previously-mentioned ways of testing it out or any others that achieve similar results, we could legitimately make at least the standard claim that we've tested what you see in the tests and the library supports what you see in the test results of CI builds.
Trimming is just too complex to claim support with anything less and, if we exempt the whole assembly, we would need to explicitly say that is what's being done, and that run-time behavior is unknown and potentially unstable.

If it sounds like too much, the simplest thing to do, at the end of the day, is to just not say anything at all about trimming. Though a CYA for people filing issues about it could be as simple as something to the effect of "we don't know," or something a wee bit more verbose like:

While some incompatibilities with assembly trimming using the .net 8.0 SDK have been addressed to varying degrees, trimming of the Terminal.Gui assembly or its transitive dependencies is untested and not guaranteed to work to any extent, for any component, unless explicitly noted and documented otherwise in that component.

Or we just don't say anything at all about trimming and fall back on the MIT license clause that no guarantee of fitness for any purpose is made.

I don't know how you intend to implement these build configurations. If you can do this and keep the examples as simple as possible, so much the better.
I'm just trying hard to make trim work with both sigle-file and Aot. By the way, what does R2R mean?
For now I know that it is not possible to have single-file and Aot configurations simultaneously. That's why I created a new project "NativeAot" to test this configuration. For now, single file is working with win-x64 platform. On other platforms it gives an error but I can't debug it due to the error that occurs when trying to attach it to the debugger, as you already know and mentioned in the README. Typically, I use the following code to attach the application to the debugger before the Application.Init call:
 while (!Debugger.IsAttached)
 {
 Thread.Sleep (500);
 }

 Debugger.Break();

Using the SelfContained project:
With the win-x64 platform it works well and I can connect to the debugger.
With the linux-x64 platform, it gives an error and I can't attach it to the debugger, because it gives the error "Unable to start debugging. Failed to attach to process: Unknown Error: 0x80131c3c"
Using the NativeAot project:
Both with the win-x64 or linux-x64 platforms they don't work well, with the assertions failing and commenting them displays the screen in black and white and I can't attach them to the debugger.

It's done with a single ItemGroup with the first element excluding all view tests, which looks like <Compile Remove="ViewTests/*.cs" /> (or whatever path the tests live under), and then one for each view type test file with a condition for the configuration (or even just a build variable - doesn't even need to be a whole configuration). Those look like <Compile Include="ViewTests/TableView.cs" Condition=" '$(ViewTypeUnderTest)' == 'TableView' " />.
To run them, when you call dotnet test, you just do this: dotnet test SingleFile.csproj -e ViewTypeUnderTest=TableView.
That then just gets parameterized for the CI runs so they still all happen at once. In github, that's just a matrix, or it can even be done entirely in the msbuild files or compiler response files. Super easy and many ways to achieve it.
As part of the build improvement issue I'm working on, I'm actually doing similar things to some of the conditionally compiled code we have to clean things up and make it easier to invoke at will.

Actually... It can be even simpler than that...
A single inclusion after the exclude, and it would look similar to <Compile Include="ViewTests/$(ViewTestCodeFile)" />, and you'd just give the file name to the command line, so you don't even need to have a separate include for every test file.

MSBuild is extremely flexible and powerful. Consequently, it may initially be daunting to approach. But, once you get the hang of a few simple common tasks like this, suddenly you turn into SuperBuilder. 😅


Identify shaders getting compiled

@brianosman is landing a series of CLs in Skia to add human-readable descriptions of shader keys. There is a new three-argument version of the store() virtual on the PersistentCache interface. We should apparently switch to overloading that version in Flutter's cache.
Ideally we'd output this information in the trace events for #76250.

Update: The work to enable this is being tracked in https://github.com/flutter/flutter/issues/78500

We think this work is no longer relevant, as our progress on Impeller means that we will need no tooling supported identification of shader usage.


Suspending server causes failed server deletion

Server  Info
Linux 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

PHP 7.0.17-2+deb.sury.org xenial+1 (cli) (built: Mar 15 2017 09:21:55) ( NTS )

Panel 0.5.7
Daemon 0.3.7 (Latest: 0.3.7)

Scenario
I created a test node, and a test server on that node (it was a ts server). After I was done testing I wanted to delete the server. I suspended the server, Idk I figured it might be safer for it to be suspended when I try to delete, I then click the delete (not force delete) option. We looked the server was still there. We figured maybe it was on a delay. So then I looked and saw the 10 minute timer. 10 minutes passed, still there so I attempted to force delete. So, I just figured I would look into it later, and worked on something else. Then the next morning while testing the API call for account creation from my E-commerce platform I noticed the accounts weren't being created. So started looking into it. Ended up restarting the worker/supervisor and then noticed the account was created. So I figure the worker got hung up on trying to delete the server or something. So that night I was wanting to test more but I didn't want to create a bunch of stuck servers. So i looked into manually deleting the previous one. Starting with trying to remove the from the DB.
Tried to truncate the servers, table and received:
> ERROR 1701 (42000): Cannot truncate a table referenced in a foreign key constraint (panel.allocations, CONSTRAINT allocations_assigned_to_foreign FOREIGN KEY (assigned_to) REFERENCES panel.servers (id))

Then tried to truncate allocation table and received:
> ERROR 1701 (42000): Cannot truncate a table referenced in a foreign key constraint (panel.servers, CONSTRAINT servers_allocation_foreign FOREIGN KEY (allocation) REFERENCES panel.allocations (id))

SS's
https://i.gyazo.com/1b42f5585784880e1e9a5b6120c22360.png
https://i.gyazo.com/28aad58c780aef029c4043ef442e3119.png
https://i.gyazo.com/4774f5a499a13b05748a7e5369a3c584.png
Additional
Force deleting the server without suspending prior works fine.
https://i.gyazo.com/64eab6d3b793a10aa750163c91448d76.png
If you would like me to do any additional testing let me know.

Did you test deleting multiple times? You said you had to restart supervisor at some point. Are you sure it got stuck because of the suspended server? I'd assume the server just didn't get deleted because of the supervisor not running correctly.

The scenario describes the first time I tried it. I went back and was able to replicate it the next night. Using the same methodology. Fresh pterodactyl database, added node, added server, suspend server, then delete server. If you would like any further information about the environment, let me know and I'll see what I can do about providing it.

I'm tempted to just scrap that queued deletion system and go back to immediate deletion.

closed by https://github.com/Pterodactyl/Panel/commit/536865b22a80c2277cd3ff8d6b777c1d7131e582


Complete Training Code and Guidelines

Would you please add complete training code and scripts and guidelines ?

Training code can be found at the repo of DINet as we use the same pre-trained model just renamed to pre-trained_lipsick.pth
Doesn't have loss convergence charts though so will need to implement that or you will be guessing when to stop each time.


mmap() failed: [12] Cannot allocate memory

Running, "php artisan geonames:install" on a homestead installations eventually runs out of memory. PHP memory usage increases until all available memory is consumed. Not sure is this a system configuration issue or an issue with the alternate names command.
Split file #289 of 2583 has 5000 rows.
Inserting alternate names 1359/5000 [▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░░░░]  27%
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
PHP Fatal error:  Out of memory (allocated<PHONE_NUMBER>) (tried to allocate 16384 bytes) in /vagrant/dr-services/vendor/laravel/framework/src/Illuminate/Database/Connection.php on line 330
mmap() failed: [12] Cannot allocate memory
mmap() failed: [12] Cannot allocate memory
PHP Fatal error:  Out of memory (allocated<PHONE_NUMBER>) (tried to allocate 65536 bytes) in /vagrant/dr-services/vendor/laravel/framework/src/Illuminate/Auth/SessionGuard.php on line 18

I was experiencing a similar issue using WampServer during the alternate names import. The culprit was "DB::enableQueryLog();" on line 389 of the AlternateNames.php file mentioned above. The query logging was generating memory for each record created until it reached critical failure. Commenting out/removing line 389 fixed the issue.


GCS failures with latest version

Hopefully just needs a minor refactor.  Note that PSQ also uses Datastore (transparently) to return task results.
It's broken with:
gapic-google-cloud-datastore-v1 (0.15.3)
google-cloud-datastore (1.4.0)
proto-google-cloud-datastore-v1 (0.90.4)
But works with:
gapic-google-cloud-datastore-v1 (0.15.3)
google-cloud-datastore (1.0.0)
grpc-google-cloud-datastore-v1 (0.14.0)
proto-google-cloud-datastore-v1 (0.90.4)
[INFO] Task failed with exception: [request() got an unexpected keyword argument 'data']
[[INFO] Writing /var/tmp/1511642221-817a08dc1d334e4587df1739da220996-PlasoTask/worker-log.txt to GCS path output/1511642221-817a08dc1d334e4587df1739da220996-PlasoTask/worker-log.txt
DEBUG: traceback.format_exc(): Traceback (most recent call last):
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 312, in run_wrapper
result = self.run(evidence, result)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/plaso.py", line 49, in run
ret, result = self.execute(cmd, result, save_files=[plaso_log], close=True)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 257, in execute
result.close(success=True)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 114, in close
self.save_local_file(logfile)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/workers/init.py", line 177, in save_local_file
new_path = writer.write(file_)
File "/usr/local/google/home/aaronpeterson/src/turbinia/turbinia/output_writers.py", line 176, in write
blob.upload_from_filename(file_, client=self.client)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 992, in upload_from_filename
size=total_bytes)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 946, in upload_from_file
client, file_obj, content_type, size, num_retries)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 867, in _do_upload
client, stream, content_type, size, num_retries)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 700, in _do_multipart_upload
transport, data, object_metadata, content_type)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/resumable_media/requests/upload.py", line 97, in transmit
retry_strategy=self._retry_strategy)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
func, RequestsMixin._get_status_code, retry_strategy)
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
response = func()
File "/usr/local/google/home/aaronpeterson/src/turbinia-old-pubsub-env/local/lib/python2.7/site-packages/google_auth_httplib2.py", line 198, in request
uri, method, body=body, headers=request_headers, **kwargs)
TypeError: request() got an unexpected keyword argument 'data'
[INFO] Task failed with exception: [request() got an unexpected keyword argument 'data']

Probably related to:
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3736

FWIW, same error with 1.2.0

Turns out this is likely only GCS (I guess I had confused some of the datastore package names), and the packages that are (were?) having dependency problems are:
google-cloud
google-cloud-core
google-cloud-storage
requests
Hopefully we can resolve these issues without breaking our pinning of the google-cloud-pubsub package to 0.26.

Stub script to reproduce this:
"""
#!/usr/bin/env python
from google.cloud import storage
file_ = '/tmp/test.txt'
open(file_, 'w').write('testing')
c = storage.Client(project='MYPROJECT')
b = c.get_bucket('MYBUCKET')
blob = storage.Blob('output/test.txt', b)
blob.upload_from_filename(file_)
"""

So, it appears that pinning the pubsub version is what is causing problems (indirectly).  When removing the pubsub pinning, the above code runs just fine.  Below is the pip freeze diff between two virtualenvs setup with the only difference being the pubsub version being pinned to 0.26.  I think the google-cloud-core package is the issue, and is mentioned in the bug referenced above.
$ diff ~/pip.head.freeze.txt ~/pip.pubsub-0.26.freeze.txt
24a26

google-auth-httplib2==0.0.3
28c30
< google-cloud-core==0.28.0

google-cloud-core==0.25.0
36c38
< google-cloud-pubsub==0.29.2

google-cloud-pubsub==0.26.0

Ugh, now after pinning google-cloud-core>=0.28.0, something is wrong with psq.  Looks like something in the pubsub library?  It seems to be the correct version though:
"""
$ pip list | grep google-cloud
gapic-google-cloud-datastore-v1 (0.15.3)
gapic-google-cloud-error-reporting-v1beta1 (0.15.3)
gapic-google-cloud-logging-v2 (0.91.3)
gapic-google-cloud-pubsub-v1 (0.15.4)
google-cloud (0.31.0)
google-cloud-bigquery (0.28.0)
google-cloud-bigtable (0.28.1)
google-cloud-core (0.28.0)
google-cloud-datastore (1.4.0)
google-cloud-dns (0.28.0)
google-cloud-error-reporting (0.28.0)
google-cloud-firestore (0.28.0)
google-cloud-language (1.0.0)
google-cloud-logging (1.4.0)
google-cloud-monitoring (0.28.0)
google-cloud-pubsub (0.26.0)
google-cloud-resource-manager (0.28.0)
google-cloud-runtimeconfig (0.28.0)
google-cloud-spanner (0.29.0)
google-cloud-speech (0.30.0)
google-cloud-storage (1.6.0)
google-cloud-trace (0.16.0)
google-cloud-translate (1.3.0)
google-cloud-videointelligence (1.0.0)
google-cloud-vision (0.28.0)
proto-google-cloud-datastore-v1 (0.90.4)
proto-google-cloud-error-reporting-v1beta1 (0.15.3)
proto-google-cloud-logging-v2 (0.91.3)
proto-google-cloud-pubsub-v1 (0.15.4)
$ pip list | grep requests
requests (2.18.4)
$ python -c 'import psq'
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/buildtools/current/sitecustomize/sitecustomize.py", line 152, in SetupPathsAndImport
return real_import(name, globals, locals, fromlist, level)
File "/lib/python2.7/site-packages/psq/init.py", line 24, in
from .broadcast_queue import BroadcastQueue
File "lib/python2.7/site-packages/psq/broadcast_queue.py", line 18, in
from google.cloud import pubsub
File "lib/python2.7/site-packages/google/cloud/pubsub/init.py", line 30, in
from google.cloud.pubsub.client import Client
File "lib/python2.7/site-packages/google/cloud/pubsub/client.py", line 21, in
from google.cloud.pubsub._http import Connection
File "lib/python2.7/site-packages/google/cloud/pubsub/_http.py", line 25, in
from google.cloud.iterator import HTTPIterator
ImportError: No module named iterator
"""

Ok, it seems like this file got moved in this PR:
https://github.com/GoogleCloudPlatform/google-cloud-python/pull/3770
And then the version is updated to 0.27.0, so I guess I need to try to pin 0.26.0.  It breaks with 0.25.0 though, so I don't have high hopes.


Arun Teltia added his json file

Fixes something #345

Welcome to Code for Cause @ArunTeltia :tada:
One correction, Your PR description should include the changes you've pushed.


Replace the goto in http.go with helper functions

Quoting /u/go255 ([link](https://www.reddit.com/r/golang/comments/67blza/commento_an_open_source_lightweight_and/dgqjxe8/

And I'm not a fan of your use of goto end in the http.go file and it's web handlers. You could had easily rewritten:

end:
result.Comments = comments
json, _ := json.Marshal(result)
w.Header().Set("Access-Control-Allow-Origin", "*")
fmt.Fprintf(w, "%s", string(json))

Into a helper function and call it/return instead of relying on the goto.

How about creating a result receiver method that writes the response:
type resultContainer struct {
        Success  bool      `json:"success"`
        Message  string    `json:"message"`
        Comments []Comment `json:"comments,omitempty"`
}

func (res *resultContainer) render(w http.ResponseWriter) {
        if res == nil {
                res = &resultContainer{false, "Some internal error occurred", nil}
        }
        w.Header().Set("Access-Control-Allow-Origin", "*")

        json, err := json.Marshal(res)
        if err != nil {
                w.Write([]byte(`{"Success":false,"Message":"Internal Server Error"}`))
                return
        }
        w.Write(json)
}


Reset training websites database daily

[ ] set up an automated job to restore the training website to its initial state
[ ] document a process for doing this manually (in case PHI is accidentally uploaded)

Possible solution:
could create a script on the psql-connect bastion host that truncates the db tables periodically on a corn via psql

Reached out to Sparkle to see what the current needs are for the training DB and waiting on her response to decide how to proceed. Specifically, is just recreating the DB and redeploying the app good enough?


Find a proper place for registering SAXMessages native resource bundle

In https://github.com/apache/camel-quarkus/pull/1793/files#diff-a2ef5a4c98491936f1d708b2ecd89661R83 we register com.sun.org.apache.xerces.internal.impl.msg.SAXMessages in our FOP extension. It does not feel like the right place. I guess it should go to our or Quarkus JAXP extension.

Please, assign to me.

As a part of Fop extension, there was registered bundle com.sun.org.apache.xerces.internal.impl.msg.SAXMessages. Bundle is not necessary for proper functionality but it helps in case there is an exception.
Example: Real problem is missing reflection registration for some classes, but developer will see only that bundle can not be loaded.
I'd like to add this bundle into camel-quarkus jaxp component, because it can help in future.

@BuildStep
NativeImageResourceBundleBuildItem initBundles() {
return new NativeImageResourceBundleBuildItem(
"com.sun.org.apache.xerces.internal.impl.msg.SAXMessages");
}

See also https://github.com/quarkusio/quarkus/issues/12129

Solved via https://github.com/quarkusio/quarkus/pull/12144 , esp. https://github.com/quarkusio/quarkus/pull/12144/files#diff-d7d3c24be2a44a5a8077b76e7b3290c6R36
I have filed https://github.com/apache/camel-quarkus/issues/1806 for the stuff we should do after Quarkus 1.9


Fix fixer tests of object-literal-key-quotes

PR checklist

[ ] Addresses an existing issue: #0000
[x] New feature, bugfix, or enhancement

[x] Includes tests

[ ] Documentation update
[x] Enable CircleCI for your fork (https://circleci.com/add-projects)

What changes did you make?
Fix test failures introduces by #1953
Is there anything you'd like reviewers to focus on?
(optional)

@ajafff thanks!

@ajafff thanks!


key not found: "TIMEZONE" (KeyError)

How to fix?
root@back:~/peatio-1.9.23# ruby -v
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
root@back:~/peatio-1.9.23# rails -v
Rails <IP_ADDRESS>
root@back:~/peatio-1.9.23# bundle exec rails server -b <IP_ADDRESS>
Traceback (most recent call last):
11: from bin/rails:8:in <main>' 	10: from bin/rails:8:in require'
9: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands.rb:17:in <top (required)>' 	 8: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:39:in run_command!'
7: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:75:in server' 	 6: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:75:in tap'
5: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:78:in block in server' 	 4: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:78:in require'
3: from /root/peatio-1.9.23/config/application.rb:17:in <top (required)>' 	 2: from /root/peatio-1.9.23/config/application.rb:18:in module:Peatio'
1: from /root/peatio-1.9.23/config/application.rb:35:in <class:Application>' /root/peatio-1.9.23/config/application.rb:35:in fetch': key not found: "TIMEZONE" (KeyError)
root@back:~/peatio-1.9.23#

root@back:~/peatio-1.9.23# ruby -v
ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
root@back:~/peatio-1.9.23# rails -v
Rails <IP_ADDRESS>
root@back:~/peatio-1.9.23# bundle exec rails server -b <IP_ADDRESS>
Traceback (most recent call last):
11: from bin/rails:8:in <main>' 	10: from bin/rails:8:in require'
9: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands.rb:17:in <top (required)>' 	 8: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:39:in run_command!'
7: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:75:in server' 	 6: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:75:in tap'
5: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:78:in block in server' 	 4: from /usr/local/rvm/gems/ruby-2.5.3/gems/railties-<IP_ADDRESS>/lib/rails/commands/commands_tasks.rb:78:in require'
3: from /root/peatio-1.9.23/config/application.rb:17:in <top (required)>' 	 2: from /root/peatio-1.9.23/config/application.rb:18:in module:Peatio'
1: from /root/peatio-1.9.23/config/application.rb:35:in <class:Application>' /root/peatio-1.9.23/config/application.rb:35:in fetch': key not found: "TIMEZONE" (KeyError)
root@back:~/peatio-1.9.23#

From Peatio README:
Things You Should Know
RUNNING AN EXCHANGE IS HARD.
This repository is not a turn key solution and will require engineering and design of security process by your company, with or without our assistance. This repository is one component among many we recommend using for composing an enterprise grade exchange. It is highly recommended to deploy a UAT environment and build automated tests for your needs, including Functional tests, Smoke tests and Security vulnerability scans. You may not need to have an active developer on Peatio source code, however, we recommend the following team setup: 1 dev/ops, 3 frontend developers (react / angular), 2 QA engineers, 1 Security Officer.
SECURITY KNOWLEDGE IS A REQUIREMENT.
Peatio cannot protect your customers if you leave your admin password 1234567, or open sensitive ports to public internet. No one can. Running an exchange is a very risky task because you're dealing with money directly. If you don't know how to make your exchange secure, hire an expert.
You must know what you're doing, there's no shortcut. Please get prepared before you continue:

Rails knowledge
Security knowledge
Cloud and Linux administration
Docker and Kubernetes administration
Micro-services and OAuth 2.0

Maybe you should ask questions here: https://discuss.rubykube.io/


Missing details about having multiple subscriptions / vnets

MSFT still positions subscriptions to be one of the scale units and even recommends it down at the application or business group level (e.g. https://azure.microsoft.com/en-us/blog/organizing-subscriptions-and-resource-groups-within-the-enterprise/).  This structure then forces you into having many virtual networks.  Please add more realistic examples of how you would structure a network architecture across a global organization with multiple subscriptions (and therefore vnets) across multiple azure regions.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fec640d8-1064-c083-4313-5e8c23fbc2cf
Version Independent ID: 5f7c63ca-7ab2-9129-4c34-191286e67af1
Content: Connect an on-premises network to Azure - Azure Reference Architectures
Content Source: docs/reference-architectures/hybrid-networking/expressroute-vpn-failover.md
Service: architecture-center
Sub-service: reference-architecture
GitHub Login: @telmosampaio
Microsoft Alias: pnp

@danmyhre For connectivity between VNETs, VNET Peering is the feature that links them together.
This is mentioned in the Azure Enterprise Scaffold doc (under Core Network) that is referenced in the link that you've shared.

Right, so what I suggesting is that more realistic large enterprise examples be added to the detailed network topology design documentation.  That is is what is missing.  Get into the details about remote gateways, transit networks, UDRs, etc, etc

@danmyhre Thanks for the feedback! We are assigning this issue to the content author for further review.

AB#160241 - Thanks for reporting - this issue is under review

For the most part, this is recommended best practices. There have been some updates and this material stays accurate in CAF: https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/govern/guides/standard/


'langmap': the comma (,) character requires not one, but two backslashes to be escaped

To Reproduce

Run gvim.exe --clean --cmd "language messages English" --cmd "set encoding=utf-8".
Execute :set langmap=б;\,. gVim emits E357: 'langmap': Matching character missing for <0431>.
Execute :set langmap=б;\\,, no errors are emitted.

Environment:

Vim version 8.2.1815
OS: Windows 10 Home 1803
Terminal: GUI.

See :help option-backslash.  The backslash needs to be escaped for a literal backslash to be in the actual option value.

Thanks for the help, @jamessan, and in #7101 too.
What confuses me here is the semicolon (;), also being a special character for 'langmap', doesn't require a second backslash, i.e. :set langmap=ж;\;, for example, works.


Issue with installation of Ionic v2 - No Remote GIT

Short description of the problem:
When i install with ionic CLI v2.1.4, i get the following issue i have shown in screenshot. Im not sure where i go wrong. Im a newbie into Ionic Framework. PFA error.

same problem here after npm install -g ionic@latest
here is log:
npm-debug.zip

@kovack i hope someone will help us really soon. I too have the same issue with ionic2 framework.

Could you please try this again? We recently made changes to the app-base and this should no longer be an issue for you.  Below is the commit that fixed the issue.
driftyco/ionic-storage@92f9032d97f32959e6ae6642bd4d37070792faf3

thanks jthoms1
now it's working


Floating = true

In the code for thermal generator, you'll find floating = true:

This allows the thermal generators to be placed on slag, which is why I was surprised when the Magmatic Generator could not be.
This PR allows the Magmatic Generator to be placed on slag.

No, As i said in other PR, its too OP


can it  run by milvus instead of local file，which separate service by database

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Hello, we are working on offering connectors to different data storages. We will soon (after Christmas holidays 😊) publish an update on this.


Feature

Linters added. My first portfolio project. For mobile nav only for now.

Thank u so much. I'll do so. But I'd like to know how do I tag you?
On Wed, May 25, 2022, 10:38 PM TufoinNkuo @.***> wrote:

@.**** requested changes on this pull request.
Hi @HtetNaing0814 https://github.com/HtetNaing0814 ,
While you made a great effort in this project, unfortunately, I cannot
proceed to review your code.
Invalid Code Review Request
You have submitted a project that does not have linter checks. you can see
how to setup linters here
https://github.com/microverseinc/linters-config/tree/master/html-css
Your Code Review Request will be marked as invalid in your Dashboard, so
please submit a new one once you are ready 🙏
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something
is not 100% clear.
Please, remember to tag me in your question so I can receive the
notification.
Please, do not open a new Pull Request for re-reviews. You should use the
same Pull Request submitted for the first review, either valid or invalid
unless it is requested otherwise.
Invalid Code Review Request does not count into the code reviews limit.
—
Reply to this email directly, view it on GitHub
https://github.com/HtetNaing0814/Portfolio/pull/1#pullrequestreview-985105689,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASORG4ALJYXNVWI23Y6JCXDVLZF7PANCNFSM5W5HV22Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

I do really have no idea what you are talking about. I thought I added
linters before submitting again. Furthermore, Do I need to make new pull
request?
On Thu, May 26, 2022, 2:27 PM Syeda Juveria Afreen @.***>
wrote:

@.**** commented on this pull request.
Invalid Code Review Request ❌
Hi @HtetNaing0814 https://github.com/HtetNaing0814,
While you made a great effort in this project, unfortunately, I cannot
proceed to review your code.
You have submitted a project, that doesn't have linters setup correctly
which means i cant see the green check marks, doesn't have a pull request
description here
https://github.com/HtetNaing0814/Portfolio/pull/1#issue-1248146496
which means it is empty and make sure to add a descriptive pull request
title like your project name "Portfolio: Setup".
If you are not able to understand kindly tag me as shown below.
Your Code Review Request will be marked as invalid in your Dashboard, so
please submit a new one once you are ready 🙏
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something
is not 100% clear.
Please, remember to tag me @.*** https://github.com/sja-thedude]
in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the
same Pull Request submitted for the first review, either valid or invalid
unless it is requested otherwise.
Invalid Code Review Request does not count into the code reviews limit.
—
Reply to this email directly, view it on GitHub
https://github.com/HtetNaing0814/Portfolio/pull/1#pullrequestreview-985860889,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASORG4BTBQMKUV7CKAATXNLVL4VHNANCNFSM5W5HV22Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

Somebody help me please.
On Thu, May 26, 2022 at 4:18 PM Htet Naing @.***> wrote:

I do really have no idea what you are talking about. I thought I added
linters before submitting again. Furthermore, Do I need to make new pull
request?
On Thu, May 26, 2022, 2:27 PM Syeda Juveria Afreen <
@.***> wrote:

@.**** commented on this pull request.
Invalid Code Review Request ❌
Hi @HtetNaing0814 https://github.com/HtetNaing0814,
While you made a great effort in this project, unfortunately, I cannot
proceed to review your code.
You have submitted a project, that doesn't have linters setup correctly
which means i cant see the green check marks, doesn't have a pull request
description here
https://github.com/HtetNaing0814/Portfolio/pull/1#issue-1248146496
which means it is empty and make sure to add a descriptive pull request
title like your project name "Portfolio: Setup".
If you are not able to understand kindly tag me as shown below.
Your Code Review Request will be marked as invalid in your Dashboard, so
please submit a new one once you are ready 🙏
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if
something is not 100% clear.
Please, remember to tag me @.*** https://github.com/sja-thedude]
in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use
the same Pull Request submitted for the first review, either valid or
invalid unless it is requested otherwise.
Invalid Code Review Request does not count into the code reviews limit.
—
Reply to this email directly, view it on GitHub
https://github.com/HtetNaing0814/Portfolio/pull/1#pullrequestreview-985860889,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASORG4BTBQMKUV7CKAATXNLVL4VHNANCNFSM5W5HV22Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

I do really have no idea what you are talking about. I thought I added linters before submitting again.

@HtetNaing0814 You have to correctly set up linters, which means i can't see the green check marks. Also, your PR doesn't have a pull request description https://github.com/HtetNaing0814/Portfolio/pull/1#issue-1248146496 which means it is empty, kindly add a description of what changes you made in the project. Also, make sure to add a descriptive pull request title like your project name is "Feature", kindly replace it with "Portfolio: Setup".


Make WriteTest more resilient to Randomness

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

In the worst case scenario for random key assignment in
Write.ApplyShardingKey, the chance of the number of records per output
shard was too high. This makes the test significantly less likely to
flake.

R: @dhalperi

LGTM, will merge once green.

According to math, this should only flake 8x10^6 probability.
Goodness, dragging that out of my brain was like being interviewed.


Exception (maybe in Ray?) during Hyperion processing

We recently did an exercise in which we ran ISOFIT on about 3,000 scenes from the Hyperion instrument. We’ve encountered a vexing problem with Ray in over 10% of the runs. Here’s a sample of the end of the log:
(_run_chunk pid=32211) row 1463/3472, (1463/1736 local), 50.66 spectra per second
(_run_chunk pid=32211) row 1464/3472, (1464/1736 local), 163.46 spectra per second
(_run_chunk pid=32212) row 3296/3472, (1560/1736 local), 46.29 spectra per second
*** SIGTERM received at time=1648857994 on cpu 42 ***
PC: @     0x2aaaaacdade2  (unknown)  pthread_cond_timedwait@@GLIBC_2.3.2
@     0x2aaaaacde630  (unknown)  (unknown)
@     0x2aaabb3c3837        384  ray::core::CoreWorkerMemoryStore::GetImpl()
@     0x2aaabb3c409c         64  ray::core::CoreWorkerMemoryStore::Get()
@     0x2aaabb3c42ad        208  ray::core::CoreWorkerMemoryStore::Get()
@     0x2aaabb34f614        368  ray::core::CoreWorker::Get()
@     0x2aaabb26dd57        240  __pyx_pw_3ray_7_raylet_10CoreWorker_31get_objects()
@     0x55555568259b  (unknown)  method_vectorcall_VARARGS_KEYWORDS
@     0x5555558b2440  (unknown)  (unknown)
[2022-04-01 17:06:34,440 E 79502 79502] logging.cc:321: *** SIGTERM received at time=1648857994 on cpu 42 ***
[2022-04-01 17:06:34,440 E 79502 79502] logging.cc:321: PC: @     0x2aaaaacdade2  (unknown)  pthread_cond_timedwait@@GLIBC_2.3.2
[2022-04-01 17:06:34,451 E 79502 79502] logging.cc:321:     @     0x2aaaaacde630  (unknown)  (unknown)
[2022-04-01 17:06:34,451 E 79502 79502] logging.cc:321:     @     0x2aaabb3c3837        384  ray::core::CoreWorkerMemoryStore::GetImpl()
[2022-04-01 17:06:34,451 E 79502 79502] logging.cc:321:     @     0x2aaabb3c409c         64  ray::core::CoreWorkerMemoryStore::Get()
[2022-04-01 17:06:34,451 E 79502 79502] logging.cc:321:     @     0x2aaabb3c42ad        208  ray::core::CoreWorkerMemoryStore::Get()
[2022-04-01 17:06:34,451 E 79502 79502] logging.cc:321:     @     0x2aaabb34f614        368  ray::core::CoreWorker::Get()
[2022-04-01 17:06:34,451 E 79502 79502] logging.cc:321:     @     0x2aaabb26dd57        240  __pyx_pw_3ray_7_raylet_10CoreWorker_31get_objects()
[2022-04-01 17:06:34,451 E 79502 79502] logging.cc:321:     @     0x55555568259b  (unknown)  method_vectorcall_VARARGS_KEYWORDS
[2022-04-01 17:06:34,464 E 79502 79502] logging.cc:321:     @     0x5555558b2440  (unknown)  (unknown)
The occurrences appear to be non-deterministic, i.e., if I re-run a scene that had such a failure, it will generally run to completion. However, I think that they all occur during the empirical line inference. We are using systems with no GPUs.
Any idea what might be causing this problem, or how we might address it?
Thanks,
-PT

Great to hear about bulk Hyperion processing!  This error sounds frustrating though, thanks for posting.
It's a little hard to say for sure...but the fact that it occurs only in the empirical line - and that it's nondeterministic - makes me think that it's some form of IO issue...only because the write-rates there are high.  What kind of system are you runninng on?  One possibility - depending on the system particulars - might be to set output directories to scratch space on local nodes, and then to do a single-threaded copy back to the main storage disk after the fact.
The only other thing I can think of is that you're running out of memory, and something like cgroups is stepping in to kill the job....but this is a bit of a weird spot in the code to run out of memory, so I'm not sure that's right.  Again, some specs might help for thinking this through.
I can say that I haven't seen this during bulk processing on either of two HPC systems that we frequently use, nor have I heard it reported from folks that utilize AWS - but I'm not sure there's been large enough scale processing there to confirm.
If anyone else has seen this, please speak up!


automatic update of remaining of rust-sgx-sdk

Some files have been copied from rust-sgx-sdk while the crates are handled with cargo.
Make sure that the files in the folder edl and rust-sgx-sdk are updated to the same hash as the crates.

Done with commit d002d07f2.
The makefile determines if the local files are older than the HEAD of rust-sgx-sdk and downloads the required files.


Parallel upload?

Baidu upload is super slow, is uploading chunks in parallel possible?

Valid suggestion and I think this is possible, but pity that this is not a small change and I can't tell if / when this will be implemented...

Tracked under #128