Benchmark bulk entity create API

Questions about bulk upload limits:

What is the upper limit of an entity CSV that backend can handle?
How does that change with # of rows vs. # of columns?
What limits will be imposed from other parts of the system (e.g. 100MB nginx request body limit ) or limits of what frontend can handle?
Frontend might be able to parse a larger file than backend?

Optimization idea:
There was an optimization using sql.unnest made to another part of backend for inserting many form fields at once. Should that approach be used here, too?  https://github.com/gajus/slonik?tab=readme-ov-file#inserting-large-number-of-rows
Commit where this was added to insertMany: https://github.com/getodk/central-backend/commit/222b2a85501f68e614c587d2081a28dc2c8a938e
What is the upper limit?

Postgresql because database supports only 65535 parameters

10 parameters per entity def --> maybe ~6K entities able to be inserted at once without the unnest optimization? Things to try!!
Related idea:
Should createMany and createNew (multiple vs. single entity creation) be combined? Does it make sense to use the multi case for a single entity or is createNew better optimized for the single-entity scenario?

I have some basic CSV parsing in Frontend in place, and I've been benchmarking that parsing. I also tried sending the parsed entities to Backend, and so far, I'm seeing two limits.
I'm able to send 6553 entities, but there's a Postgres error for 6554 entities. That's consistent with the limit mentioned above. Happy to share my JSON data if that'd be helpful!
Secondly, even though client_max_body_size is 100m in nginx (here), Backend itself seems to set a much smaller limit for JSON request bodies, of 250 kB. I had to increase that limit in order for larger requests to go through.
https://github.com/getodk/central-backend/blob/f98187d76307dafe76482a882beba16854008c9f/lib/http/service.js#L33

I just ran into the 6553 max entity and 250kb JSON limit myself.
Removing that limit, I've so far been able to upload 6553 entities with 1000 properties, but not 1500 properties.

Based on discussion yesterday, we're hoping to be able to bulk-create 50,000 entities with 20 properties each, where each value is 100 characters.

250kb JSON limit

This seems like a good idea to have in general. Can we lift/override the limit only for this endpoint?

100MB nginx request body limit

Two thoughts about the nginx limit:

50,000 entities x 20 properties/entity x 100 bytes/property = 100 MB of data, which doesn't include JSON syntax characters (e.g., quotes and commas) or property names that are repeated for each entity. I think that means that we should increase the nginx limit, though probably just for this one endpoint. What should we increase it to?
Currently, before Frontend attempts to send a file to Backend (e.g., a form attachment), it checks whether it exceeds the nginx limit, showing an informative error message if it does. If we think there's a risk of the new nginx limit being exceeded, I think we should consider having Frontend do something similar for request bodies that aren't files. (Frontend will send JSON, not the file itself, so the current check won't be tripped.) Right now, Frontend would just say, "Something went wrong: error code 413."

I think that means that we should increase the nginx limit

We discussed this today and concluded that it'd be fine to leave the nginx limit where it is today, at 100 MB. We've seen that the size of the JSON request body is usually larger than the CSV file, so this will probably limit the size of CSV files to less than 100 MB. (I think the JSON is larger in large part because certain strings are repeated for each entity: label, data, and each property name. The CSV file only lists property names once.) At some point, we may make the nginx limit configurable (related: getodk/central#609).

Currently, before Frontend attempts to send a file to Backend (e.g., a form attachment), it checks whether it exceeds the nginx limit, showing an informative error message if it does. If we think there's a risk of the new nginx limit being exceeded, I think we should consider having Frontend do something similar for request bodies that aren't files.

I've filed getodk/central#610 for this idea, but for now, we're just going to implement getodk/central#609. I've filed that as a separate issue so that we can continue to use this issue to discuss other limits and aspects of benchmarking.

I just wanted to leave a comment with some observations from @alxndrsn on Slack. @alxndrsn uploaded a CSV file of 100K entities:

from drag-drop -> preview is near-instant
"sending file" about 25s (counting in my head)
"processing file" about 90s (counting in my head)
as a ballpark, it takes less than 1 second to insert this data into my local db

It'd be interesting to learn more about why processing the file takes so long.


wasm version

Do you have plan to release wasm version?
I want to try to use it, if it will be released.

I don't have any specific plans to release a wasm version, however this library should be usable if you're integrating into a larger wasm-based C++ application.
If you'd like to use this library on the web, I'd recommend using the base pixelmatch library from https://github.com/mapbox/pixelmatch. Javascript VMs are quite fast, and the algorithm is the same as pixelmatch-cpp17, so I think both should have about the same performance.


Improve docs to run Alluxio in cluster

break previous page to run Alluxio cluster into two pages, one for single master and one for HA
individual improvements in each page

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4019/
Test PASSed.

@apc999 Could you address the merge conflicts please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4406/
Test PASSed.

@maobaolong thanks for the suggestions! I am trying to address them in the next round (in a week or two) -- this PR is large enough simply splitting the original one doc to two with writing correction and improvement.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4445/
Test PASSed.

@calvinjia  PTAL, plus one question regarding worker-side master add left

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4489/
Test PASSed.

ping @calvinjia

@apc999 looks good overall, one comment left

@calvinjia  comments addressed with rolling restart added

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4538/
Test PASSed.

alluxio-bot, merge this please.

alluxio-bot, cherry-pick this to branch-2.0 please.


Miscellaneous and improvements and fixes

Coverage increased (+1.4%) to 90.909% when pulling d8a40dcb0f857c6f5725fbd1e704a5dd21470d07 on testtube_improvements into 9a35cdb5391b625f7ebc849e2b61983b9bb456f7 on master.

Coverage increased (+1.4%) to 90.909% when pulling 3a3f39b38211379cd75ea73850729b8896df38cc on testtube_improvements into 9a35cdb5391b625f7ebc849e2b61983b9bb456f7 on master.


usage as a library

Describe the feature:
I would like to be able to use goss as a package from within other go applications, there are various infra automation scenarios where its desirable to deliver the system as a single binary.
I'd like to do an integration into choria.io and some other things
Describe the solution you'd like
At least I would like to be able to do a validate, a few of the functions in validate.go, store.go, template.go and a few others do os.Exit() and require a number of unexported functions.
Would you be open to adjusting things a bit so a external package can use goss as a package for at least doing Validate?
Similar to #112
I'd roughly keep things are, even accepting a cli context in validate but I'd add a few helpers to let me do:
err = goss.ValidateWithOptions(goss.GossFile(...), goss.VarsFile(...), goss.OutputFormat(goss.JSON))

additional options for pkg type, output format and options, color, duration, retry duration and max concurrency
Describe alternatives you've considered
I tried to write this from the outside - alas, the os.Exit() etc is a pain

This is a reasonable request and to be honest, I am not a fan of using os.Exit() and cli.Context outside of the CLI functions.

additional options for pkg type, output format and options, color, duration, retry duration and max concurrency

This is confusing, could you go into more detail of your use case, please?
My only worry is that @aelsabbahy will not prioritize this highly. Then you could work on this and I would love to help out.

This is a reasonable request and to be honest, I am not a fan of using os.Exit() and cli.Context outside of the CLI functions.

additional options for pkg type, output format and options, color, duration, retry duration and max concurrency

This is confusing, could you go into more detail of your use case, please?

simply they are the other options the package today get from the CLI - I will give equivelant in code.

Marking this as approved. It's very much an improvement to the current code. I probably won't dedicated personal time to this due to other issues being higher priority for me. That said, I will gladly review and merge changes that introduce this with testing.
Honestly, the parts of the code that will benefit most from this refactor are the parts that would benefit most from having proper go based tests in place.

This is a reasonable request and to be honest, I am not a fan of using os.Exit() and cli.Context outside of the CLI functions.

In all seriousness, there's so much of the codebase that I've wanted to rewrite. My time has been limited and I end up favoring bug fixes and features to codequality improvements.
I'm a huge fan of this ticket.

I am also quite busy so my effort will be to get this done with as little as possible to reach my needs. Hopefully others can pick up where I left off.

Work being done:

[ ] remove os.Exit() WIP https://github.com/aelsabbahy/goss/pull/546
[ ] remove cli.Context WIP https://github.com/aelsabbahy/goss/pull/546
[ ] remove log.Fatal

@ripienaar Is this done at this point?
Also, what was your use-case, curious on where goss is being used as a library =)

Yup done thanks.
I make choria.io, my use case is to make a very large scale dgoss like thing and some other stuff
Choria will have a new monitoring related distribution and could run goss checks regularly and handle distribution of the check definitions etc
And things like adhic checks etc. It’s a orchestration system known to be deployed on fleets with 6 to 7 number node counts

Awesome, thanks!
I'll read up on choria, sounds like an interesting project =)

$ choria req scout goss_validate file=/etc/choria/goss.yaml
....
Summary of Failed Tests:

   0: 27

Summary of Success:

   5: 27

Summary of Tests:

   5: 27

Finished processing 27 / 27 hosts in 3.279081845s

It sends back the reports etc, summaries show that all 27 machines was sucesful, ran 5 tests had no failures.
So 3 seconds is kind of slow - but these machines are in euro, us and uk.  On a lan with fast networks 27 000 machines would take the same time :P


Updates

Hi @nshmyrev,
Some minor things here:

Add .gitignore
Add gradle wrapper
Update/format readme
Bump android sdk to 25

Cheers

I'm sorry, I do not think it is a good idea to include gradle wrapper into repository, it is autogenerated anyway.

I saw the folder structure.  Using gradle wrapper version gives structure. Nevertheless, I have little knowledge with NDK to know if it can be optimized and use the wrapper instead of the machine's global gradle. So for now I reverted gradle wrapper commit.

Ok, thank you!


Removed reset generators

:see_no_evil:

Codecov Report

Merging #13792 (bdda952) into main (21219ff) will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #13792      +/-   ##
==========================================
- Coverage   56.31%   56.27%   -0.04%
==========================================
  Files         553      553
  Lines       45992    45992
  Branches     4128     4126       -2
==========================================
- Hits        25902    25884      -18
- Misses      20047    20065      +18
  Partials       43       43

Impacted Files
Coverage Δ

core/server/data/db/connection.js
55.00% <0.00%> (-28.34%)
:arrow_down:

core/shared/config/loader.js
93.42% <0.00%> (-3.95%)
:arrow_down:

core/shared/config/utils.js
84.09% <0.00%> (-2.28%)
:arrow_down:

core/server/models/base/plugins/events.js
70.95% <0.00%> (+1.47%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21219ff...bdda952. Read the comment docs.


Support for relative links

Fixes #25

Thank you for the contribution and your patience @Hi-Fi
Thank you for the help with the code review @lukiffer


Added basic support for ip6tables, with a set of basic rules

Following issue #8, added a set of basic rules for IPv6 and some templates and documentation. Updated the test script for IPv6 and added a check on INPUT policy.
Tested and passed with Vagrant build.

Thanks!


Fix col description in prod6 layout files

Thanks for the corrections!
There are already subfolders (even for prod1!), but I didn't find time to clean this up (and change also the analysis scripts, which do look in the main directory for the files).


fix: spoof UA header for Github Releases

(I realize this is an absurd thing to have to do)
It seems as though GIthub Releases is blocking at least some portion of urllib request User Agents. We're noticing this with python3.11 exclusively. We've seen the failures on 3.11.1-3.11.4, running on the latest and the previous OSX version on both Apple Silicon and Intel machines as well as running on Debian instances in CircleCI. I've not yet found any official announcement from Github of them trying to do this on purpose.
I also noticed that once I spoofed the UA from my machine, Github starting allowing non-spoofed requests to work as well, which makes this even more absurd. Using a non-default header always makes it succeed, using the default header only succeeds if you've already gotten a successful response from Github Releases for that object (I'm guessing the User-Agent check is avoided if the object is cached?).
Trying to install from the latest main:
$ python3.11 -m pip install .
Processing /private/var/folders/7h/c97966217hx9pp1hsgd_0v200000gq/T/tmpenv.6uosr92z/shellcheck-py
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: shellcheck-py
  Building wheel for shellcheck-py (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for shellcheck-py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [4 lines of output]
      running bdist_wheel
      running build
      running fetch_binaries
      error: HTTP Error 503: Backend unavailable, connection timeout
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for shellcheck-py
Failed to build shellcheck-py
ERROR: Could not build wheels for shellcheck-py, which is required to install pyproject.toml-based projects

Spoofing the UA header to basically anything else makes it all work.
$ python3.11 -m pip install .
Processing /private/var/folders/7h/c97966217hx9pp1hsgd_0v200000gq/T/tmpenv.6uosr92z/shellcheck-py
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: shellcheck-py
  Building wheel for shellcheck-py (pyproject.toml) ... done
  Created wheel for shellcheck-py: filename=shellcheck_py-<IP_ADDRESS>-py2.py3-none-macosx_13_0_arm64.whl size=6502105 sha256=4b129d267217c2230a21c17f2006bf12f7e2aa02ae4ddb715044da36db8d6a49
  Stored in directory: /Users/kevin/Library/Caches/pip/wheels/0a/d1/86/e85b5c337e3c6916aff444b1425b27cb53362b62e47f45ec78
Successfully built shellcheck-py
Installing collected packages: shellcheck-py
Successfully installed shellcheck-py-<IP_ADDRESS>

503 is gateway timeout -- are you sure you didn't just hit github during an outage?

As far as I can tell, this issue is continuing to affect me and my team this week. Also verified with several other folks over the past week that applying this solution solved their problems: most recent news on that was yesterday and I am not aware of any Github API issues yesterday.
That said, there certainly could be some unreported-upon issues going on at Github right now, but I've yet to see a 503 with the faked UA.

I'd recommend you contact github support

Fair enough, we can handle this as a fork.


web service handler with []*api.Endpoint option

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
Some times we may use one micro api gateway for rpc and http at the same time.
Can we add a endpoints ...*api.Endpoint option for go-micro/web/Service interface Handle() and HandleFunc() func.
My implement
So that user can use web/Service start a http service for micro api gateway. Otherwise users need to implement it themselves.
Just an idea to see if it's necessary to add to the go-micro/web
e.g.
package main

import (
	"fmt"
	"net/http"

	"github.com/hb-go/micro-plugins/web"
	"github.com/micro/go-micro/api"
	"github.com/micro/go-micro/util/log"
)

func main() {
	// create new web service
	service := web.NewService(
		web.Name("go.micro.api.console.web"),
		web.Version("latest"),
	)

	// initialise service
	if err := service.Init(); err != nil {
		log.Fatal(err)
	}

	service.Handle("/console/", &handler{}, &api.Endpoint{
		Name:    "console",
		Host:    []string{"localhost:8080"},
		Path:    []string{"^/console"},
		Method:  []string{"POST", "GET", "DELETE"},
		Handler: "proxy",
	})

	// run service
	if err := service.Run(); err != nil {
		log.Fatal(err)
	}
}

type handler struct {
}

func (*handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	w.Write([]byte(fmt.Sprintf("request success, path: %v", r.URL.Path)))
}

Additional context
Add any other context or screenshots about the feature request here.

this is already covered by handler rpc, does we need also proxy handler support for this?

web service != api service
we're working on a model of consolidation, we just don't have an answer yet


[FEATURE] Mood board integration

Re: University Monday 2023/09/04 / 2023-09-07
4 Misho,
4.1 Please start to look at Rokmetro Vogue:
Look at the Vogue source Stephen shared with you first.
Then look at the new Rokmetro vogue source.
Get the Vogue App and look at Mood Board, this is what JP wants you to make, but a better improved version.
4.2 If we supply you a single Figma document with assets/flow/etc will that be enough for the new Mood board UI/UIX specification?
Fwd: Boarding Figma Files / 2023-09-13
---------- Forwarded message ---------
From: Peter Robertson<EMAIL_ADDRESS>Date: Tue, 12 Sept 2023 at 11:33
Subject: Boarding Figma Files
To: Paul, John<EMAIL_ADDRESS>Clementine Levingston<EMAIL_ADDRESS>Hi JP
Here is the link https://www.figma.com/file/KYJjsf3gEHaT6W6e3OrkMW/Vogue-Runway-Jan-Release---Specifications?type=design&node-id=169%3A25767&mode=design&t=Y1IlYsSDYztrNd6c-1 to the boarding figma files. The elements we discussed today are all prefaced with Boarding - and then the name of the feature in the pages column of Figma. I've also added a PDF (some of the formatting might be a bit odd) to this mail.
I have only included the additional & new boarding features here, and have marked on all the screens which ones use the current implementation in the app, which are updated and which are new.
Let me know if any issues with access to the file for your team and any questions I can answer.
Many Thanks,
Peter
Runway Boarding Specifications.pdf

Created in a wrong repo, closing.


feat: Add request approval

I have made few changes to the request approval code. Please check and merge.


How to make the MusicService to be a remote process

I Found if I set the MusicService to be a remote process, MediaBrowserCompat.SubscriptionCallback.onError() will be called.  So, how to make it works? Thanks.

Can you try again with the code that's in master? When I move MusicService to another process it works fine as far as I can see.
<service
    android:name=".MusicService"
    android:process=":service"
    android:enabled="true"
    android:exported="true"
    tools:ignore="ExportedService">

    <intent-filter>
        <action android:name="android.media.browse.MediaBrowserService" />
    </intent-filter>
</service>

It works, Thank you.

Can you try again with the code that's in master? When I move MusicService to another process it works fine as far as I can see.


update in DPO raise several problems...

get_hh was removed. There is no dataset now.

from trl.commands.cli_utils import DpoScriptArguments, init_zero_verbose, TrlParser: No module named 'trl.commands'

Those changes are currently only on main, did you install TRL from source?


afterAll() does not run after all fixtures have been torn down

The scaladoc says:
https://github.com/scalameta/munit/blob/64fee6586a51ba1670872825f25a0d77837a3adb/munit/shared/src/main/scala/munit/Suite.scala#L39-L40
But this doesn't seem to be the case in https://github.com/typelevel/munit-cats-effect/pull/208#discussion_r901038577.

This seems like a deliberate change in https://github.com/scalameta/munit/pull/430 based on the diff to the FixtureOrderFrameworkSuite. @olafurpg can you confirm this was intended?

Aha, apologies for noise. This is specifically covered in the release notes under "breaking changes".
https://github.com/scalameta/munit/releases/tag/v1.0.0-M1


NEW INDEX:  band ratio applied to mineral exploration

Hello, friends, I have a question.
I am adding several mineral band ratios locally in my machine.
Should I add the tag mining in this part of the src/utils.py, or I leave it to you?
class IndexType(Enum):
    """
    IndexType supported by SpectralIndex DataClass
    """

    VEGETATION = "vegetation"
    WATER = "water"
    BURN = "burn"
    SNOW = "snow"
    SOIL = "soil"
    URBAN = "urban"
    KERNEL = "kernel"
    RADAR = "radar"

For my indexes, I'd add MINING = "mining". Should it be fine?

I have two other issues:

A lot of the ratios use Aster satellite. This means that the generic band indication won't fit this satellite. How should we proceed?
Many ratios I was able to find the references. However, others I can't find. Is it a problem?

Hi @rodreras!
Thank you for your messages. There are many things to solve before adding the indices:
First message:

Can we add the indices to the "soil" application domain?

Second message:

At the moment there is no standard for Aster in ASI. We are working on a unified way to accept different sensors. So, for the moment, we can't include Aster indices. I will let you know when this is possible.
All indices in ASI must have a corresponding reference to be added to the catalogue.
A composition is not considered an index. Ratios can be added as indices.

Thank you for your attention and let me know if you have more questions! For now, feel free to add indices that already meet the requirements ;)


Showing this to initial users

Is there a way that I can only show these tooltip views to a first time user? so that when they user uses the app the second time, the views are no longer there?

let launchedBefore = UserDefaults.standard.bool(forKey: "launchedBefore")
if launchedBefore {
print("Not first launch.")
} else {
print("First launch, setting UserDefault.")
UserDefaults.standard.set(true, forKey: "launchedBefore")
}
Customize the above to dismiss!!

Yes, you can do that but your issue has nothing to do with the library. Store a boolean in UserDefaults  and this way you know if your user was presented with the tooltips or not. Cheers!


Robotics Console can be used to emag check cyborgs

Title, mostly. Liltenhead included this in his video so it got a lot of visibility, so it's bound to get abused.
 On emagged cyborgs, plopping out the brain fails, so the button does not gray out and the cyborg is effectively confirmed to be emagged.
This is most likely an oversight and unintended behavior, considering the destroy button mimics an emagged cyborg being destroyed.

This is intended look at the console pr

This is intended look at the console pr

It is intended that the brain doesnt plop out but disable should have the same behavior as the normal one for consistency

Looking at the PR itself, this behavior does not feel intended. The "Destroy" functionality on emagged cyborgs pretends like the cyborg is destroyed and even removes said borg from the console altogether. I don't see why the other option would provide more feedback.
And to be honest, even if it is intended, we should try to cut the number of instant emag checks as much as possible.

yeah ill make it report removed brain in a bit so cant validcheck borgs remotely

This is intended look at the console pr

It is not intended, if it were “destroy” would not remove the cyborg from the list or feign a “cyborg has been destroyed” message


Touchpoints modal in app doesn't maintain focus

Area or Component
in app - any page
Quick Summary
The modal for Touchpoints doesn't hold focus, so screenreaders can read outside it and users can tab outside the modal.
See this same issue for the static site here: https://github.com/CDCgov/prime-simplereport/issues/4030
How to Replicate

Open Touchpoints in the SR app
Start tabbing backwards
See that you go back to the main page content

Accessibility Violation
Able to browse outside modal with screen reader: https://cdc-devtools.dequecloud.com/rule-help/en/reading-order-browse-outside-modal
Keyboard focus is not maintained in modal: https://cdc-devtools.dequecloud.com/rule-help/en/focus-modal-moves-outside
Action Requested
Ensure that keyboard focus can't escape the Touchpoints modal.

Moving this back to the backlog but we may want to icebox/close this depending on the designs Kenny comes up with. For more info: https://github.com/CDCgov/prime-simplereport/pull/4310


Add project ideas

Description of  the change
Describe more about the change.
Domain of Docs

[] Readme
[] Solidity , Html, Css , JavaScript , Web3.js / Ether.js
[] Solidity , ReactJs /NextJs , Web3.js /Ether.js
[] Solidity , Python3 , Web3.py
[] Other , feel free to mention below

Are you contributing under any open-source program ?

please assign this issue to me

what u want  to do ? please decsribe your work ?

Add detailed articles on topics like
Decentralized Blog
Building your own blog doesn't only improve your coding skill, but your online presence too. If you have a blog and share content regularly, you can get a lot of visitors which can increase your online presence.
You can use this opportunity to build your blog on any blockchain.
Checkout Nader's article on how to build a full-stack blog application with Next.js and Solidity on Polygon blockchain https://blog.suhailkakar.com/13-projects-you-can-do-to-become-a-web3-and-blockchain-master-beginner-to-expert

i gues for the hacktoberfest don't do this  this may    spam and can exclude my repo . Better  if  u made changes in the  code .
Hope u understand @akshat-jn-crypto


Can normal users access the Admin Workbench via a unique id?

Describe the bug
"THIS IS NOT A BUT. JUST A QUESTION OF FUNCTION"
If I create a workbench in the Admin workspace, the end user will not be able to see the list. However, normal user can access and modify the page by entering url directly via the generated workbench id (eg. App / v2 / workbench / a7b68ce0-d7ea-4202-9ceb-83e158861d53)
To Reproduce
Steps to reproduce the behavior:

Create workbench in admin workspace and exact workbench URL(app/v2/workbench/a7b68ce0-d7ea-4202-9ceb-83e158861d53)
Share the unique id to normal user
Normal user can access and modify workbench

Expected behavior
I wonder if this behavior was intended (a design concept for sharing)
Screenshots

Desktop (please complete the following information):

Tmap Dashboard Alpla version

Additional context
Add any other context about the problem here.

This is a permission error in workspace. I think we need to check this out as a whole. Thank you for your bug report.

Malformed..
This situation can not occur in normal, but it happend by your settings.
Your personal 'Admin workspace' set to 'default permission schema'.
I dont know why and how did you set this permission scheme, but I assume it may acted for testing.
For maintance, I close this issue and if you have any other questions, you can reopen it at any time.

Malformed..
This situation can not occur in normal, but it happend by your settings.
Your personal 'Admin workspace' set to 'default permission schema'.
I dont know why and how did you set this permission scheme, but I assume it may acted for testing.
For maintance, I close this issue and if you have any other questions, you can reopen it at any time.

I attach related.


Two bug fixes and some cleaning

While reading code I usually try to clean it. This PR is result for some cursory reading of the repo.
It also fixes two bugs in lockstep array:

new constructor didn't check that capacities match
is_empty had it's logic inverted.

Everything looks good. Thank you for the contribution!


Do not use Span Processor to calculate standard/live metrics

Still work in progress

Codecov Report

Merging #103 into master will decrease coverage by 0.87%.
The diff coverage is 80.85%.

@@            Coverage Diff             @@
##           master     #103      +/-   ##
==========================================
- Coverage   92.73%   91.85%   -0.88%
==========================================
  Files          21       21
  Lines        1142     1191      +49
  Branches      141      149       +8
==========================================
+ Hits         1059     1094      +35
- Misses         55       70      +15
+ Partials       28       27       -1

Impacted Files
Coverage Δ

...itor/sdk/auto_collection/metrics_span_processor.py
70.00% <50.00%> (-12.50%)
:arrow_down:

...ure_monitor/sdk/auto_collection/request_metrics.py
76.92% <78.84%> (+1.11%)
:arrow_up:

..._monitor/sdk/auto_collection/dependency_metrics.py
93.25% <83.33%> (-6.75%)
:arrow_down:

.../src/azure_monitor/sdk/auto_collection/__init__.py
100.00% <100.00%> (ø)

...nitor/sdk/auto_collection/live_metrics/__init__.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e9e669f...275bc84. Read the comment docs.

It would be good to know which integrations will be tracked with the given patching for requests and http.server, and the ones that aren't tracked so we can create issues for them (since we are not using spanprocessor anymore).

It would be great if you can create an issue to track which dependencies we still need to track for outgoing/incoming requests.

Created issue #106 for future improvement of this


[VERBOSE-2:shell.cc(1004)] The 'customer_io' channel sent a message from native to Flutter on a non-platform thread

Getting this error when app is starting:
[VERBOSE-2:shell.cc(1004)] The 'customer_io' channel sent a message from native to Flutter on a non-platform thread. Platform channel messages must be sent on the platform thread. Failure to do so may result in data loss or crashes, and must be fixed in the plugin or application code creating that channel.

Hey, @Anderzzon thank you for reaching out, can you please let us know how you are initializing the SDK?

Closing due to inactivity, feel free to re-open if needed.

I have the same issue. My init code in main.dart:
await CustomerIO.initialize(
config: CustomerIOConfig(
siteId: "[my_site_id]",
apiKey: "[my_api_key]",
region: Region.eu,
//config options go here
autoTrackDeviceAttributes: true,
enableInApp: true,
logLevel: CioLogLevel.debug,
),
);

@BenBach did it just start occuring? Did you update the version number etc?
Would be able to share more information so I can try to replicate it, as in flutter/dart version, CIO SDK version etc?
@cmbellman does it still occur for you.

Thanks for your continued support everyone.
Version 1.5.1 of the Flutter SDK has been deployed and includes a fix for this error message.
Please, give this new release a try. If you continue to receive this error message after install, send us a message with updated reproduction steps and we'll continue to look into it.
Thank you so much!


Micro Speech Example Deploy to STM32F746 error!

Following the method to build this example,
https://github.com/tensorflow/tflite-micro/issues/475#issuecomment-915649543
error reported like the following:
xx@xxx:~/code/tflite-micro/test/mbed-os$ mbed compile -m DISCO_F746NG -t GCC_ARM -D CMSIS_NN
[mbed] Working path "/home/lmk/code/tflite-micro/test/mbed-os" (library)
[mbed] Program path "/home/lmk/code/tflite-micro/test"
[mbed] WARNING: Missing Python modules were not auto-installed.
The Mbed OS tools in this program require the following Python modules: future, click
You can install all missing modules by running "pip install -r requirements.txt" in "/home/lmk/code/tflite-micro/test/mbed-os"
On Posix systems (Linux, etc) you might have to switch to superuser account or use "sudo"
Building project test (DISCO_F746NG, GCC_ARM)
Scan: test
Compile [  0.2%]: micro_model_settings.cc
Compile [  0.3%]: command_responder.cc
Compile [  0.3%]: main.cc
Compile [  0.4%]: audio_provider.cc
Compile [  0.5%]: no_micro_features_data.cc
Compile [  0.6%]: feature_provider.cc
Compile [  0.7%]: yes_micro_features_data.cc
Compile [  0.8%]: no_30ms_audio_data.cc
Compile [  0.8%]: yes_30ms_audio_data.cc
Compile [  0.9%]: micro_speech_model_data.cc
Compile [  1.0%]: recognize_commands.cc
Compile [  1.1%]: RTX_Config.c
Compile [  1.2%]: no_1000ms_audio_data.cc
Compile [  1.3%]: cmsis_os1.c
[Fatal Error] base.h@38,10: cstdint: No such file or directory
[ERROR] In file included from ../third_party/flatbuffers/include/flatbuffers/string.h:20,
from ../mbed-os/cmsis/CMSIS_5/CMSIS/RTOS2/RTX/Library/cmsis_os1.c:27:
../third_party/flatbuffers/include/flatbuffers/base.h:38:10: fatal error: cstdint: No such file or directory
38 | #include
|          ^~~~~~~~~
compilation terminated.
[mbed] ERROR: "/usr/bin/python3" returned error.
Code: 1
Path: "/home/lmk/code/tflite-micro/test/mbed-os"
Command: "/usr/bin/python3 -u /home/lmk/code/tflite-micro/test/mbed-os/tools/make.py -D CMSIS_NN -t GCC_ARM -m DISCO_F746NG --source .. --build ../BUILD/DISCO_F746NG/GCC_ARM"
Tip: You could retry the last command with "-v" flag for verbose output

Were you able to resolve this? I am running into this exact issue (missing cstdint). I'm hoping to get the example working and update the README with the appropriate steps once I understand all the setup required.

Were you able to resolve this? I am running into this exact issue (missing cstdint). I'm hoping to get the example working and update the README with the appropriate steps once I understand all the setup required.

I didn't solve it. Please ask questions in the tensorflow mainline, not in the micro branch.
https://github.com/tensorflow/tensorflow/issues

This issue is happening because mbed tries to compile it's C files with the string.h file of flatbuffers instead of the libc string header. It then fails to include cstdint because this is a C++ header. I resolved this issue by appending fb_ to all flatbuffer header files except for flatbuffers.h and flexbuffers.h with the shell script below. I hope this will help someone running into the same issue.
for f in *; do
	if [[ "$f" == "flatbuffers.h" || "$f" == "flexbuffers.h" ]]; then
		continue
	fi

	mv "$f" "fb_$f"
	sed -i "s@#include \"flatbuffers/$f@#include \"flatbuffers/fb_$f@g" *
done


add support for Amazon Linux 2

there is no mapping for Amazon Linux 2 so far.
I checked the name against the latest Amazon Linux 2 AMI ami-a9d09ed1 in us-west-2

exact duplicate of #65


Failed to resolve: com.github.anboralabs:spatia-room:0.1.0

I cant install this library.
Could you help me?

@mcauto were you able to solve the issue?


CT-15061 new method loadWidgets

Widget Loading On Demand!

Added a convenience method to load widgets anytime - method is under the CrowdTwist namespace.
Updated bootstraping to use the new loadWidgets()

@pjuckiewicz
@bertrandvidal
@fongstanley
@jtaoct
@chunpi
@pranaypatil
@georgehlim

:+1:


Bad link in readthedocs

In https://vulnerablecode.readthedocs.io/en/latest/introduction.html#how-does-it-work
there is this statement
These data sources (see complete list here) include
but if you click on the link, you discover that the referenced page does not exist.

Assigning to @TG1999  but available to anyone who has access to an official list of data sources.

@DennisClark @TG1999 I think I've located the missing SOURCES.rst file directly under the project/repo root rather than under /docs -- see https://github.com/nexB/vulnerablecode/blob/main/SOURCES.rst.  To be certain my "fix" (details below) is what we want, I'd like to show it to you via a huddle whenever it's convenient and make sure this now links to the data that we want to link to.  Please ping me whenever you'd like to see this.
We already incorporate this file, in its current location, in vulnerablecode/docs/source/reference_importer_overview.rst with the syntax .. include:: ../../SOURCES.rst.  Based on some googling, the problem with the missing link seems to be that because the file is outside of /docs, a link rather than an include does not work.
I've fixed this by creating a new .rst under /docs named importers_link.rst that contains an include to the SOURCES.rst file, a title, and an internal ref, like this:
.. _importers_link:

Importers
=========

.. include:: ../../SOURCES.rst

That's the entire file, and we link to it like this:
These data sources (see complete list :ref:`here <importers_link>`) include security advisories published by Linux and BSD distributions . . . .

makes sense to me  --  thanks @johnmhoran

Thanks @DennisClark .  @TG1999 If this looks good to you as well, please let me know and I'll close this as fixed.

Done and closing.


Split off Shell into separate bundle

This is part of the work to reduce the size of Filer (see #575).  Our Shell is pretty big, and not always needed.  This splits it out into a separate bundle, and if you want it, you include it.  Here's the breakdown of our build now:
$ npm run build

><EMAIL_ADDRESS>build-filer /Users/humphd/repos/filer/package.json
> parcel build --global Filer src/index.js --out-file filer.min.js --detailed-report

✨  Built in 432ms.

dist/filer.map    347.85 KB    76ms
dist/filer.js     192.61 KB    91ms
✨  Built in 237ms.

dist/filer.min.map                                  288.99 KB    56ms

dist/filer.min.js                                    74.06 KB    65ms
├── src/filesystem/implementation.js                 21.46 KB    21ms
├── node_modules/buffer/index.js                     19.54 KB    17ms
├── src/filesystem/interface.js                       3.84 KB    16ms
├── lib/intercom.js                                   3.76 KB     9ms
├── src/providers/websql.js                            2.5 KB    18ms
├── src/providers/indexeddb.js                        2.21 KB    17ms
├── node_modules/path-browserify/index.js              2.1 KB    23ms
├── node_modules/es6-promisify/dist/promisify.js       1.8 KB    21ms
├── src/node.js                                        1.7 KB     6ms
├── node_modules/process/browser.js                   1.69 KB    23ms
└── + 19 more assets

><EMAIL_ADDRESS>build-shell /Users/humphd/repos/filer/package.json
> parcel build --global Shell src/shell/index.js --out-file shell.min.js --detailed-report

✨  Built in 182ms.

dist/shell.map    107.9 KB    34ms
dist/shell.js     72.74 KB    33ms
✨  Built in 175ms.

dist/shell.min.map                                  100.17 KB    30ms

dist/shell.min.js                                    24.01 KB    30ms
├── node_modules/minimatch/minimatch.js               7.32 KB     9ms
├── src/shell/index.js                                3.88 KB     5ms
├── node_modules/path-browserify/index.js              2.1 KB     7ms
├── node_modules/brace-expansion/index.js             2.07 KB     5ms
├── node_modules/es6-promisify/dist/promisify.js       1.8 KB    10ms
├── node_modules/process/browser.js                   1.69 KB     8ms
├── src/constants.js                                  1.65 KB     3ms
├── src/errors.js                                     1.01 KB     6ms
├── lib/async.js                                        795 B     9ms
├── node_modules/balanced-match/index.js                619 B     1ms
└── + 3 more assets

We'll need to use a major version bump for this, since it's a breaking change.  I won't land this until I get the update stuff in and working in #640.

Codecov Report

Merging #643 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #643      +/-   ##
==========================================
- Coverage   86.56%   86.55%   -0.02%
==========================================
  Files          16       16
  Lines        1742     1740       -2
==========================================
- Hits         1508     1506       -2
  Misses        234      234

Impacted Files
Coverage Δ

src/filesystem/interface.js
93.28% <ø> (-0.09%)
:arrow_down:

src/shell/index.js
88.32% <ø> (ø)

src/index.js
100% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bfe4385...5d62fbc. Read the comment docs.


docs: add sitemap

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (6faba8c) 79.68% compared to head (3184963) 79.68%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #72   +/-   ##
=======================================
  Coverage   79.68%   79.68%
=======================================
  Files           3        3
  Lines         123      123
  Branches       24       24
=======================================
  Hits           98       98
  Misses         11       11
  Partials       14       14

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Lag when receiving data

I'm trying to stream some sensor data for an experiment and i created a very simple script that lets me connect and receive sensor data using this library.
The number of messages is around 100 per second, each message is very small, it's something like {"date": "xx", "rate": "xx", "direction": "xx"}, At first everything went flawlessly, but then i started noticing some discrepancies in the data i was receiving, in particular there seems to be some "lag" where some messages are received after some seconds while other messages are lost.
Now i checked everything and i'm sure the problem is in the part where i receive the data through my websocket connection, so i was wondering if maybe the big amount of messages per second can create problems and if so, how to avoid them. My code is pretty simple, for now i'm just receiving the data and printing it:
async def connect():
	while True:
		try:
		     async with websockets.client.connect('MYURL') as ws:
		             while True:
				    message = ''

				    try:
					msg = await asyncio.wait_for(ws.recv(), 5)
					message = json.loads(msg)
					print(message)

				    except asyncio.TimeoutError:
					break
		except Exception as e:
			logging.exception(e)

This code should make sure to keep the connection alive when it breaks and to restart it when i have not received data for five second. Am i doing something wrong? Is there any way to fix this?

Somewhat related to #414.
In very old versions of websockets, there used to be a bug where canceling recv() could cause a message to be dropped, but I'd be surprised if you didn't use a recent enough version.
100 messages / second on a single connection shouldn't be a problem. I have experimental evidence that websockets can handle tens of thousands of messages on thousands of connections.
I don't have enough info to reproduce or diagnose the issue. Given that WebSocket runs over TCP it's extremely unlikely that messages would be lost, except if you're disconnecting and reconnecting, in which case it's expected.
When you say:

Now i checked everything and i'm sure the problem is in the part where i receive the data through my websocket connection
can you clarify how you know that the TCP connection never breaks?

What exception is getting logged?
Finally, can you make sure the device you're connecting to responds to pings correctly, or alternatively set ping_timeout=None to disable disconnection detection?

Thank you a lot for your clarifications! I've been making some further checks and i found that the problems were due to drops in the connection of the server where the data was being generated, which lead me to believe (wrongly) that the problem happened when i started to stream from the websocket. Using ping_timeout=None is very helpful too. Thank you a lot!

You shouldn't need ping_timeout=None unless the server is quite shitty, but hey, maybe it is :-)

I thought it wasn't too much - but maybe it is :)

I'm sorry for reopening the issue here, but i just wanted to to understand something, so forgive me if i'm saying something stupid here. Now that i know that my problem is resource here, i investigated a bit into how does all of this work, and from what i understood, my problem might be the buffer getting filled by the flood of messages. Is it possible? And is there any possible fix to this other than getting better hardware?

If a buffer getting filled is a problem, the solution is usually to make it smaller, so that backpressure kicks in earlier.
There are some details here: https://websockets.readthedocs.io/en/stable/deployment.html?highlight=bufferbloat#buffers

Looking into this, thank you!


Migration to the new kotlinx-coroutines-test API

References
Migration to the new kotlinx-coroutines-test API

Fixed by #156


How can one know for what particular download listeners are triggered?

Hi, thank you for this awesome lib!
I have a question is there a chance to know download ID in OnStartOrResumeListener or OnProgressListener?

You will have to keep a mapping for that. Currently, the library does not provide. We can add this feature.

Or as usual, pass different download listener to different download.

Or as usual, pass different download listener to different download.

That's what I do at the moment :)


Android - Glitch occurs when scrolling through threads on the android app

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Open the android app
Scroll through the threads

Expected Result:
Experience smooth scrolling while navigation.
Actual Result:
The app experiences glitches when navigating.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[ ] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 3.57-5
Reproducible in staging?: n/a
Reproducible in production?: n/a
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/93399543/595cf6d9-9fec-459d-b3f9-31560d5bee47
Expensify/Expensify Issue URL:
Issue reported by: @jo-ui
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1689453640673329
View all open jobs on GitHub

@hungvu193 #20984 is about dissappearing messages when you scroll. This issue is about a glitch when you scroll not about disappearing messages.

I was assigned while I was OOO, I'll test soon.

I'm going to need to reopen the discussion and confirm if we should keep this one open.

Based on the conversation here and in Slack, I'm going to close.


nisdomain.erb variable typo error

line 9,
/bin/nisdomainname <%= @domain %>
missing "@"

@lkthomas I'll take a look at this in the next few days, but feel free to submit a PR.

Fixed and merged in #39.


Add missing space

Docs Build status updates of commit 688c85d:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

api/PowerPoint.AddIns.md
:bulb:Suggestion
View
Details

api/PowerPoint.AddIns.md

Line 2, Column 1: [Suggestion-description-missing] Missing required attribute: 'description'.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

@jozefizso Awesome, thanks so much! We appreciate your keen eye


The behavior and the activity log are inconsistent when deleting one folder which contains a file ends with a dot and the setting "Disable file share trailing dot support" is checked

Storage Explorer Version: 1.30.0-dev
Build Number: 20230603.6
Branch: rel/1.30.0
Platform/OS: Windows 10/Linux Ubuntu 20.04/MacOS Ventura 13.4 (Apple M1 Pro)
Architecture: x64/arm64
How Found: From running test cases
Regression From: Not a regression
Steps to Reproduce

Open Settings -> Make sure the setting 'Disable File Share Trailing Dot Support' is unchecked.
Expand one storage account -> File Shares.
Create a file share -> Create a folder named 'test'.
Upload a file which ends with a dot to the folder.
Check the setting 'Disable File Share Trailing Dot Support'.
Delete the folder.
Check whether the behavior and the activity log are consistent.

Expected Experience
The behavior and the activity log are consistent.
Actual Experience
The behavior and the activity log are inconsistent, there is a successful activity log and the folder still displays.

This issue is due to AzCopy's historical behavior in how they treat deleting files that "doesn't exist". AzCopy typically has a policy of "if what you want to delete isn't there, then that's a successful deletion". When the trailing dot flag is disabled, the dot will be dropped. AzCopy will be unable to find the file/treat it as "it doesn't exist", and therefore a successful delete.
A fix to this issue depends on AzCopy correcting this deletion behavior.


spaghetti css in two css files

You have two css files and both of them have similar css and they look really messy
Very spaghetti css every where
You could have easily separate them and provide a maintainability

I'm sorry I'm not quite sure what you mean? Are you referring to the slidebars.min.css? That's precisely the same as slidebars.css just minimized to reduce file size.

Thanks for the reply
I was referring to your css files with duplicate styles.Both of the
style.css and slidebars.css have styles for same elements. For example you
have sb slide bar in both of the css files which makes very unmanageable
codes
On 04/02/2016 7:07 PM, "Adam Smith"<EMAIL_ADDRESS>wrote:

I'm sorry I'm not quite sure what you mean? Are you referring to the
slidebars.min.css? That's precisely the same as slidebars.css just
minimized to reduce file size.
—
Reply to this email directly or view it on GitHub
https://github.com/adchsm/Slidebars/issues/215#issuecomment-179702504.

Yep that's how it is supposed to be slidebars.min.css is exactly the same as slidebars.css. You are only supposed to use one of them.
If you don't need to edit the script or are using it in a production site, use slidebars.min.css. If you need to edit it or are developing a site, use slidebars.css.

You are talking about minified version of same css file.
But I am talking about two different css files you got under same css folder
On 05/02/2016 12:35 AM, "Adam Smith"<EMAIL_ADDRESS>wrote:

Yep that's how it is supposed to be slidebars.min.css is exactly the same
as slidebars.css. You are only supposed to use one of them.
If you don't need to edit the script or are using it in a production site,
use slidebars.min.css. If you need to edit it or are developing a site, use
slidebars.css.
—
Reply to this email directly or view it on GitHub
https://github.com/adchsm/Slidebars/issues/215#issuecomment-179841562.

Which two files are you talking about then? Where are they located?

Have a look at your source of http://plugins.adchsm.me/slidebars/
you can see

slidebars.min.css and style.css both have duplicated tags
I know slidebars as been minified but once you converted to normal file you
can still see ".sb-slidebar" all over the places and spread across both of
the files. You need lots of housekeeping to clean those css
On Fri, Feb 5, 2016 at 1:16 AM, Adam Smith<EMAIL_ADDRESS>wrote:

Which two files are you talking about then? Where are they located?
—
Reply to this email directly or view it on GitHub
https://github.com/adchsm/Slidebars/issues/215#issuecomment-179860413.

The style.css file isn't part of the plugin, that's why it is not included in the download anywhere. That is only used for styling the website. Slidebars doesn't come with any styling or components, which is why it is so lightweight.

But my point is  when you  were doing your own website  you  hv used
style.css for styling. In that style.css and slidebars.css both of them
have same css elements. What  you could have done is adding parameters to
qjuery function.
Dont forget  that  people come in and check how you have done things and
wired things together before they hire you

They are kept separate so it's easy to maintain.


feat(workflow): add arm build

I disabled ARM builds in #9 because compiling Rust crates on ARM through QEMU was causing builds to take up to and sometimes over 3 hours. It's far more efficient to just do it locally with the Dockerfile in the repo, unless someone can figure out a way to optimise it, which I cba to do :eyes:


x/build/dashboard: darwin-arm-mg912baios and darwin-arm64-mn4m2zdaios builders consistently failing

Starting around https://build.golang.org/log/776c688e90aaa17f09782242647391a368eeabc7 (Apr. 9), the two darwin-arm* builders have been failing with lots of messages of the form
go_darwin_arm_exec: GOIOS_DEV_ID not set
run $GOROOT/misc/ios/detect.go to attempt to autodetect
FAIL	archive/tar	0.017s

It seems like that may be due to a change in the builder configuration, but I'm not really sure.
@eliasnaur, could you look into it (or loop in someone who can)?

Yes, I've pinged @steeve. I believe he's on it.

I am. I'll try to completely factory reset the arm64 phone. arm32 should be ok I think

Any update?

Yes, the builders seems ok now.

mg912baios seems to be unplugged (https://build.golang.org/log/532482aa5995eaeb33bbd8dbeca8064b22a11014):
No device found with udid 8e5c23a5d0843d1ffe164ea0b2f2500599c3ebff, is it plugged in?

@bcmills yes, the 32 bits device is having issues, I'm trying to get another device to test.

I bisected the problem in ideviceinstaller. The problem is https://github.com/libimobiledevice/ideviceinstaller/commit/4a7f0b01483f2b1e3c11447bb4654b181deacf41
I have opened https://github.com/libimobiledevice/ideviceinstaller/issues/119 to track it. In the mean time I'm going to revert that commit on the build machine.

I have downgraded ideviceinstaller, and the builder now seems to run ok. I'll monitor.

The builder is back online for the two archs, and now they are failing for good reasons

now they are failing for good reasons

It looks like the current failures are not sending output back to the runner. Is there some change that needs to be made to extract it from the device on failure?
Are there issues filed for the current failures?

I don't know of any way than manually running the tests. See
https://github.com/golang/go/issues/34751#issuecomment-539988570
https://github.com/golang/go/issues/34751#issuecomment-540532877

Thanks, I'll do it tomorrow!

FWIW, you might find https://github.com/golang/go/issues/34501#issuecomment-542535507 useful. You can now use TRY= to target a particular set of builders.

I couldn't reproduce the failures on an iPhone 5S running iOS 10. I ran:
GO_BUILDER_NAME=darwin-arm64-mn4m2zdaios GOARCH=arm64 CGO_ENABLED=1 CC_FOR_TARGET=$(pwd)/../misc/ios/clangwrap.sh ./all.bash

(GO_BUILDER_NAME set runs tests that normally doesn't run).
Perhaps the devices are somehow broken, although it is unlikely. I noticed that the builder test times quite a bit higher than the times for my old 5S. Compare
https://build.golang.org/log/c6e5a939431f25560b0f1b6837fe465f7c1cb0e5
with

$ GO_BUILDER_NAME=darwin-arm64-mn4m2zdaios GOARCH=arm64 CGO_ENABLED=1 CC_FOR_TARGET=$(pwd)/../misc/ios/clangwrap.sh ./all.bash
Building Go cmd/dist using /Users/elias/go1.10.1.
Building Go toolchain1 using /Users/elias/go1.10.1.

Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, darwin/amd64.
Building packages and commands for target, darwin/arm64.
Testing packages.
ok  	archive/tar	18.201s
ok  	archive/zip	11.190s
ok  	bufio	7.152s
ok  	bytes	14.369s
ok  	bytes/hash	16.824s
ok  	compress/bzip2	15.776s
ok  	compress/flate	12.629s
ok  	compress/gzip	12.045s
ok  	compress/lzw	13.847s
ok  	compress/zlib	14.690s
ok  	container/heap	16.063s
ok  	container/list	16.188s
ok  	container/ring	14.735s
ok  	context	11.305s
ok  	crypto	12.365s
ok  	crypto/aes	15.064s
ok  	crypto/cipher	14.427s
ok  	crypto/des	13.884s
ok  	crypto/dsa	15.297s
ok  	crypto/ecdsa	15.013s
ok  	crypto/ed25519	15.736s
ok  	crypto/elliptic	15.728s
ok  	crypto/hmac	14.655s
ok  	crypto/internal/subtle	11.942s
ok  	crypto/md5	11.154s
ok  	crypto/rand	13.423s
ok  	crypto/rc4	16.539s
ok  	crypto/rsa	16.583s
ok  	crypto/sha1	16.769s
ok  	crypto/sha256	14.369s
ok  	crypto/sha512	13.625s
ok  	crypto/subtle	13.943s
ok  	crypto/tls	16.474s
ok  	crypto/x509	10.810s
ok  	database/sql	18.426s
ok  	database/sql/driver	21.070s
ok  	debug/dwarf	17.096s
ok  	debug/elf	14.532s
ok  	debug/gosym	16.432s
ok  	debug/macho	16.740s
ok  	debug/pe	16.651s
ok  	debug/plan9obj	15.850s
ok  	encoding/ascii85	12.912s
ok  	encoding/asn1	12.634s
ok  	encoding/base32	12.794s
ok  	encoding/base64	11.628s
ok  	encoding/binary	11.053s
ok  	encoding/csv	13.882s
ok  	encoding/gob	13.455s
ok  	encoding/hex	17.578s
ok  	encoding/json	19.731s
ok  	encoding/pem	20.637s
ok  	encoding/xml	18.714s
ok  	errors	18.430s
ok  	expvar	15.001s
ok  	flag	14.468s
ok  	fmt	13.896s
ok  	go/ast	15.172s
ok  	go/build	15.188s
ok  	go/constant	14.788s
ok  	go/doc	14.414s
ok  	go/format	14.197s
ok  	go/importer	12.109s
ok  	go/internal/gccgoimporter	15.107s
ok  	go/internal/gcimporter	15.507s
ok  	go/internal/srcimporter	15.818s
ok  	go/parser	14.590s
ok  	go/printer	13.898s
ok  	go/scanner	12.324s
ok  	go/token	11.212s
ok  	go/types	10.876s
ok  	hash	13.154s
ok  	hash/adler32	14.959s
ok  	hash/crc32	14.631s
ok  	hash/crc64	13.114s
ok  	hash/fnv	12.926s
ok  	html	13.659s
ok  	html/template	14.707s
ok  	image	14.538s
ok  	image/color	15.613s
ok  	image/draw	16.387s
ok  	image/gif	16.972s
ok  	image/jpeg	17.097s
ok  	image/png	22.929s
ok  	index/suffixarray	22.717s
ok  	internal/cpu	15.546s
ok  	internal/fmtsort	20.220s
ok  	internal/poll	14.344s
ok  	internal/reflectlite	15.453s
ok  	internal/singleflight	12.261s
ok  	internal/trace	13.367s
ok  	internal/xcoff	13.251s
ok  	io	11.707s
ok  	io/ioutil	14.055s
ok  	log	14.236s
ok  	log/syslog	14.328s
ok  	math	12.967s
ok  	math/big	13.534s
ok  	math/bits	15.471s
ok  	math/cmplx	17.359s
ok  	math/rand	20.752s
ok  	mime	17.974s
ok  	mime/multipart	18.536s
ok  	mime/quotedprintable	17.830s
ok  	net	12.891s
ok  	net/http	31.848s
ok  	net/http/cgi	16.170s
ok  	net/http/cookiejar	36.788s
ok  	net/http/fcgi	37.614s
ok  	net/http/httptest	39.733s
ok  	net/http/httptrace	25.852s
ok  	net/http/httputil	27.925s
ok  	net/http/internal	25.935s
ok  	net/http/pprof	24.583s
ok  	net/internal/socktest	25.154s
ok  	net/mail	25.736s
ok  	net/rpc	26.676s
ok  	net/rpc/jsonrpc	23.754s
ok  	net/smtp	24.981s
ok  	net/textproto	23.249s
ok  	net/url	22.979s
ok  	os	22.862s
ok  	os/exec	18.322s
ok  	os/signal	23.432s
ok  	os/user	20.376s
ok  	path	17.529s
ok  	path/filepath	16.829s
ok  	plugin	13.654s
ok  	reflect	8.814s
ok  	regexp	13.082s
ok  	regexp/syntax	14.190s
ok  	runtime	40.523s
ok  	runtime/debug	12.701s
ok  	runtime/internal/atomic	21.586s
ok  	runtime/internal/math	50.398s
ok  	runtime/internal/sys	50.778s
ok  	runtime/pprof	61.725s
ok  	runtime/pprof/internal/profile	48.881s
ok  	runtime/trace	48.433s
ok  	sort	43.406s
ok  	strconv	25.252s
ok  	strings	23.497s
ok  	sync	13.304s
ok  	sync/atomic	15.885s
ok  	syscall	22.605s
ok  	testing	23.282s
ok  	testing/quick	24.825s
ok  	text/scanner	24.583s
ok  	text/tabwriter	13.329s
ok  	text/template	12.708s
ok  	text/template/parse	12.437s
ok  	time	14.342s
ok  	unicode	17.984s
ok  	unicode/utf16	18.413s
ok  	unicode/utf8	18.245s
ok  	cmd/addr2line	14.911s
ok  	cmd/api	16.565s
ok  	cmd/asm/internal/asm	19.435s
ok  	cmd/asm/internal/lex	22.476s
ok  	cmd/compile	14.944s
ok  	cmd/compile/internal/gc	13.900s
ok  	cmd/compile/internal/ssa	14.009s
ok  	cmd/compile/internal/syntax	17.947s
ok  	cmd/compile/internal/test	16.562s
ok  	cmd/compile/internal/types	12.867s
ok  	cmd/cover	7.833s
ok  	cmd/doc	8.944s
ok  	cmd/fix	15.593s
ok  	cmd/go	20.502s
ok  	cmd/go/internal/auth	29.320s
ok  	cmd/go/internal/cache	35.796s
ok  	cmd/go/internal/dirhash	43.076s
ok  	cmd/go/internal/generate	41.227s
ok  	cmd/go/internal/get	43.067s
ok  	cmd/go/internal/imports	43.416s
ok  	cmd/go/internal/load	42.931s
ok  	cmd/go/internal/lockedfile	42.894s
ok  	cmd/go/internal/lockedfile/internal/filelock	43.013s
ok  	cmd/go/internal/modconv	46.200s
ok  	cmd/go/internal/modfetch	46.436s
ok  	cmd/go/internal/modfetch/codehost	45.561s
ok  	cmd/go/internal/modfile	44.536s
ok  	cmd/go/internal/modload	41.002s
ok  	cmd/go/internal/module	41.762s
ok  	cmd/go/internal/mvs	41.725s
ok  	cmd/go/internal/note	40.770s
ok  	cmd/go/internal/par	39.428s
ok  	cmd/go/internal/renameio	42.728s
ok  	cmd/go/internal/search	41.736s
ok  	cmd/go/internal/semver	42.004s
ok  	cmd/go/internal/sumdb	42.078s
ok  	cmd/go/internal/tlog	41.237s
ok  	cmd/go/internal/txtar	44.945s
ok  	cmd/go/internal/web	45.697s
ok  	cmd/go/internal/work	43.101s
ok  	cmd/gofmt	36.385s
ok  	cmd/internal/buildid	27.874s
ok  	cmd/internal/dwarf	19.661s
ok  	cmd/internal/edit	16.184s
ok  	cmd/internal/goobj	14.456s
ok  	cmd/internal/obj	15.065s
ok  	cmd/internal/obj/arm64	14.677s
ok  	cmd/internal/obj/x86	13.805s
ok  	cmd/internal/objabi	14.729s
ok  	cmd/internal/src	15.072s
ok  	cmd/internal/test2json	16.841s
ok  	cmd/link	16.346s
ok  	cmd/link/internal/ld	16.424s
ok  	cmd/link/internal/sym	17.148s
ok  	cmd/nm	15.624s
ok  	cmd/objdump	14.410s
ok  	cmd/pack	15.963s
ok  	cmd/trace	14.936s
ok  	cmd/vet	13.833s
os/user with tag osusergo
ok  	os/user	12.085s
GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
ok  	runtime	50.483s
sync -cpu=10
ok  	sync	4.940s
../misc/cgo/test
lldb: running program
scatter = 0x1001d7da4
sqrt is: 0
testLibgcc is disabled on ARM64 with clang due to lack of libgcc.
testLibgcc is disabled on ARM64 with clang due to lack of libgcc.
testLibgcc is disabled on ARM64 with clang due to lack of libgcc.
testLibgcc is disabled on ARM64 with clang due to lack of libgcc.
testLibgcc is disabled on ARM64 with clang due to lack of libgcc.
hello from C
PASS
ok  	misc/cgo/test	52.168s
../misc/cgo/testso
ok  	misc/cgo/testso	53.345s
../misc/cgo/testsovar
ok  	misc/cgo/testsovar	16.533s
../misc/cgo/testcarchive
PASS
ALL TESTS PASSED
Installed Go for darwin/arm64 in /Users/elias/go-tip
Installed commands in /Users/elias/go-tip/bin

Looking at just the names of the failing tests, my hypothesis is that they are the memory-intensive ones, and that the tests are being killed by some sort of OOM signal.
That would also explain the lack of output: Go tests buffer their output until the test completes, so if the test binary is killed before it completes, it may well produce no output.

If that hypothesis is correct, then it seems to me that the ideal fix is to have the memory-intensive tests in the affected packages query (somehow‽) for the process's memory limit and invoke t.Skip if there is not enough memory to complete the test.

Do these devices already scale the test parallelism down to 1? If not, that might be another good starting point.

If the were with memory or parallelism, why can't I reproduce the problem on the modest iPhone 5S? I still believe the devices need a factory reset or replacement. @steeve ?

I am giving a a look at the device builder.
I'm trying to run the failing tests in verbose mode, I see in https://build.golang.org/log/a5ca5d49956d984e773f40a0a4c56e79dd7858de
FAIL	bytes	114.748s
FAIL	crypto/tls	72.274s
FAIL	crypto/x509	82.845s
FAIL	math/big	79.414s
FAIL	net/http	97.111s
FAIL	runtime	79.566s
FAIL	runtime/pprof	126.951s
FAIL	strings	119.247s

That said, when running on HEAD I get a weird error:
$ GOIOS_DEVICE_ID=8e5c23a5d0843d1ffe164ea0b2f2500599c3ebff GOARCH=arm CGO_ENABLED=1 go test -v archive/tar
# runtime/cgo
clang: error: argument unused during compilation: '-mno-thumb' [-Werror,-Wunused-command-line-argument]
FAIL	archive/tar [build failed]
FAIL

I got it working:
$ PATH=$(pwd)/bin:$PATH GOROOT=$(pwd) GO_BUILDER_NAME=darwin-arm-mg912baios GOARCH=arm CGO_ENABLED=1 CC=$(pwd)/misc/ios/clangwrap.sh ./bin/go test -v bytes
lldb: running program
=== RUN   TestNewBuffer
--- PASS: TestNewBuffer (0.00s)
=== RUN   TestNewBufferString
--- PASS: TestNewBufferString (0.00s)
=== RUN   TestBasicOperations
--- PASS: TestBasicOperations (0.00s)
=== RUN   TestLargeStringWrites
--- PASS: TestLargeStringWrites (0.07s)
=== RUN   TestLargeByteWrites
--- PASS: TestLargeByteWrites (0.06s)
=== RUN   TestLargeStringReads
--- PASS: TestLargeStringReads (0.00s)
=== RUN   TestLargeByteReads
--- PASS: TestLargeByteReads (0.00s)
=== RUN   TestMixedReadsAndWrites
--- PASS: TestMixedReadsAndWrites (0.01s)
=== RUN   TestCapWithPreallocatedSlice
--- PASS: TestCapWithPreallocatedSlice (0.00s)
=== RUN   TestCapWithSliceAndWrittenData
--- PASS: TestCapWithSliceAndWrittenData (0.00s)
=== RUN   TestNil
--- PASS: TestNil (0.00s)
=== RUN   TestReadFrom
--- PASS: TestReadFrom (0.00s)
=== RUN   TestReadFromPanicReader
--- PASS: TestReadFromPanicReader (0.00s)
=== RUN   TestReadFromNegativeReader
--- PASS: TestReadFromNegativeReader (0.00s)
=== RUN   TestWriteTo
--- PASS: TestWriteTo (0.00s)
=== RUN   TestRuneIO
--- PASS: TestRuneIO (0.00s)
=== RUN   TestNext
--- PASS: TestNext (0.00s)
=== RUN   TestReadBytes
--- PASS: TestReadBytes (0.00s)
=== RUN   TestReadString
--- PASS: TestReadString (0.00s)
=== RUN   TestGrow
--- PASS: TestGrow (0.01s)
=== RUN   TestGrowOverflow
--- PASS: TestGrowOverflow (0.00s)
=== RUN   TestReadEmptyAtEOF
--- PASS: TestReadEmptyAtEOF (0.00s)
=== RUN   TestUnreadByte
--- PASS: TestUnreadByte (0.00s)
=== RUN   TestBufferGrowth
--- PASS: TestBufferGrowth (0.01s)
=== RUN   TestEqual
--- PASS: TestEqual (0.00s)
=== RUN   TestEqualExhaustive
--- PASS: TestEqualExhaustive (0.26s)
=== RUN   TestNotEqual
FAIL	bytes	13.790s
FAIL

I don't see any actual test failures in the verbose output. Is the lldb wrapper somehow injecting a nonzero exit code?

@bcmills I'm editing the log in place just in case, you might want to refresh

For some reason I can't run the runtime test, it just hangs on
lldb: running program

darwin-arm64-mn4m2zdaios is now failing pretty consistently in misc/cgo/testcarchive.TestOsSignal:
2020-02-20T05:04:39-49f8d45/darwin-arm64-mn4m2zdaios
2020-02-20T03:27:46-715e8bb/darwin-arm64-mn4m2zdaios
2020-02-19T23:28:35-8fa2b6d/darwin-arm64-mn4m2zdaios
2020-02-19T22:04:29-f4f7db4/darwin-arm64-mn4m2zdaios
2020-02-19T21:53:23-8a8adc2/darwin-arm64-mn4m2zdaios
2020-02-19T21:18:17-2f2e97c/darwin-arm64-mn4m2zdaios
2020-02-18T19:44:25-a0cf2c8/darwin-arm64-mn4m2zdaios
2020-02-18T17:55:23-a9ea91d/darwin-arm64-mn4m2zdaios

@steeve this is far outside the four week failure widow specified in our porting policy, which is grounds for removal. Are you able to devote the needed time to maintain this port? Is there someone else who should take over?

Since we've already said in the 1.14 release notes:

Go 1.14 will likely be the last Go release to support 32-bit binaries on iOS, iPadOS, watchOS, and tvOS (the darwin/arm port). Go continues to support the 64-bit darwin/arm64 port.

This may be the time for us to remove the 32 bit darwin ports.
Apple started requiring 64 bit in iOS 11 (released September 2017). Is there a compelling reason for us to continue to support darwin/arm, @steeve @eliasnaur ?

Per #37611, we will be removing darwin/arm (keeping darwin/arm64) so that builder will go away soon.
As there hasn’t been a reply yet about either builder, it’s likely that we’ll remove both from the dashboard as all they’re providing is noise at the moment.

Hey guys
I think we can remove the builders. Unfortunately the builds are failing and my attempts at finding the root cause was unsucessful.
In any case, the Corellium builder is way more reliable than what we have at the moment.
If somebody is willing to help me debug it tho, I can take half a day and do so.

Apple started requiring 64 bit in iOS 11 (released September 2017). Is there a compelling reason for us to continue to support darwin/arm, @steeve @eliasnaur ?

The reason was to keep some WatchOS versions working.

Apple started requiring 64 bit in iOS 11 (released September 2017). Is there a compelling reason for us to continue to support darwin/arm, @steeve @eliasnaur ?

The reason was to keep some WatchOS versions working.

Also see #34751 where it was discussed in more detail.


Dereference Razor document state information in FAR scenarios.

In FAR scenarios with large amounts of Razor documents we generate Razor output for every document in the world and then feed that information to Roslyn to populate a set of C# buffers with Razor's generated C# documents. Prior to this we were holding onto all of the data involved in creating that generated C# document; instead we now dereference that data on the DocumentSnapshot to allow for GC to do its job. Also did this for excerpt generation to ensure we don't break scenarios where 1 C# reference would result in X number of Razor documents being re-generated and allocate loads of memory.
Added lots of comments to raise some of the reasons why we clear state in certain places.
Added tests to ensure we're clearing data in the correct places.
Added tests to ensure that the new source text wrapped source documents are 100% equivalent with what was there before.
Added a skipped test to ensure we get coverage of the excerpt stored state clearing once Roslyn unblocks us.

Results for a 2200 Razor document project:

Committed Memory
Peak Memory

Before
1.14GB
1.38GB

After
633MB
787MB

Improvement
44%
43%

This is going to go into 16.1-preview4. Have a few offline conversations on if this should be serviced or not but so far it's not looking like it should be.

🆙 📅 to use weak references all over the place 😄
Review when you can, need to get this in today.


Pull spans for cost estimates from PSCIS input spreadsheets direct

Estimates work currently as we have remade  pscis_rd.csv with 0280-extract-bcfishpass-rd-data-modelled_closed_bottom.R to reflect new span estimates built with 0340-tables-phase2-cost-estimate.R but really we should pull direct from the latest version of the submission sheets.

really we should be using the hab con widths to calculate the spans too.  Not sure if that is in the cards on this one though :smirk:


LiveShareClient: discovery UI for adding a new server to monitor shows incorrect results

The list view used in DiscoveryServer does not seem to contain 100% correct values.

The discovery is a bit unstable (server disappear and come back the next scan cycle).
A string copy issue seems to happen with the host list. The end of the string buffer is reused for the host names.
For example, the following hosts are available on the network with a running BEATs discovery server
HOST1234
HOST2

The discovery display will show:
HOST1234
HOST2234
This very 'C' like strcpy behavior is definitely a bug.

Unstable discovery not related to this bug


Update lavish bootstrap

During research into why #719 happened, I found some problems. In a recent commit I removed bootstrap.css because we were already including lavish-bootstrap.css. Lavish is a site that generates a bootstrap file based on some colors, but otherwise it is identical to the stock bootsrap. We generated our lavish from an old 3.0.X version of bootstrap, but were including bootstrap 3.3.4. This meant that any changes in bootstrap from 3.0 to 3.3.4 were not in our lavish file. Hence when we removed the stock bootstrap from the CSS includes, some things like this broke (maybe others too that we have not yet found).
The correct fix for this is to regenerate our lavish file. However I don't know the colors it was originally generated with. We could find this out with a small amount of effort.
But there's a larger problem because people have been editing our lavish-bootstrap file with random additions. Ideally our lavish-bootstrap file will remain unchanged from the generator, and any additions or changes we want to make will be in customstyles.css. This will allow us to upgrade the lavish file without stomping on the other various change we've made over time.
This is a larger effort and maybe not worth the trouble now. Just putting the issue here so we are aware of what the current state is.

A larger questions is whether we should do a large-scale refactor of our CSS, which probably has a lot of legacy and inconsistent details. @petervandivier, it'd be great if you could eventually partner with our new Prod Ops Eng to assess this.

@jseldess - typo’d @ I think

Sorry @petervandivier. I meant @anagio.

We have closed this issue because it is more than 3 years
old. If this issue is still relevant, please add a comment and reopen
the issue. Thank you for your contribution to CockroachDB docs!


Access to Year 2 Follow-up DTI and T1 data

ABCC only has baseline data. I would like to download both baseline and Year 2 follow-up which prevents me to use the nda-abcd-s3-downloader tool. I turned back to the abcd-dicom2bids tool but it does not recognise my NDA username and password 'Failed to create NDA token using the username and decrypted password from /home/eb07/.abcd2bids/config.ini.' The ndar_awskeys.txt file isn't generated. Any idea of what could be the problem here? Alternatively, what would be the best way to transform DICOM to BIDS for both baseline and Year 2 follow up fast-track ABCD images?
Many thanks.

HI @eliacello - Thanks for your email! We are currently processing the Year 2 data for the ABCC and will upload to the NDA as soon as we are able. There were some unforeseen challenges as the ABCC collection is curated to use data that has passed on the Fast Track QC document and we were waiting for updates (updates are still ongoing, so there will likely be future changes as well).
Which of the following paths would be best for you?
(1) Wait for our next release as it will be the correct data according to the Fast-Tracked ABCD QC.
(2) Start processing these data yourself. If you would like to do all of this processing yourself, I have assigned some folks from our lab to see if they have insights about the NDA token errors.

Hi @arueter1 , Thank you so much for your quick response. I guess it must be difficult to estimate but will the next release occur in 2 or more 8 months? In the latter case I  will be willing to try to start processing myself because of time constraints. I am only working with a small subset of participants (~1500) so hopefully, it won't be too time-consuming to check their QC status..? @perronea, it would be great to have your input on the issue I encounter with NDA token generator when I use the abcd-dicom2bids. Thank you very much for you help!

Hi @eliacello - We just got an updated Fast Track QC document this week with significantly more Year 2 data QC'ed. This changes our trajectory and timelines. I am guessing that we will not have our next release uploaded to the NDA for 3 months. The NDA needs to QC the upload before it is released publicly, which can take anywhere from 1-5 months (hopefully the shorter timeframe)! Given that information, I do not think we will have our next release out for 4-8 months. I hope this is helpful for you to make decisions that work best for you!
I am going to close this ticket. If you have further issues with the NDA token generator, please create an issue on the applicable repo: https://github.com/ABCD-STUDY/abcd-dicom2bids/issues


Use some fs shorthand functions.

This is just some cleanup, using some fs convenience functions that I think make the code more readable.
Note: While going through these, I found the out_dir_is_preserved test was broken and not doing what it was supposed to.

r? @Eh2406
(rust_highfive has picked a reviewer for you, use r? to override)

Otherwize the part I skimed looks good. r+ when you are ready. :-) and thanks for the cleanup.

Hm, I sorta expected rustfmt would clean that up, but apparently that was removed because it broke semantics in some cases.
@bors r=Eh2406

:pushpin: Commit 4ae79d2ffdfb1511c1d4a3e659ca1422a8be57e6 has been approved by Eh2406

:hourglass: Testing commit 4ae79d2ffdfb1511c1d4a3e659ca1422a8be57e6 with merge 85fd2865fc9cccddd7049ff45b6803c2b7ca2be8...

:sunny: Test successful - checks-azure
Approved by: Eh2406
Pushing 85fd2865fc9cccddd7049ff45b6803c2b7ca2be8 to master...


No more logs in /var/log/traffic_ops/traffic_ops.log

On the systems I upgraded to 1.1.3, seems like there is no more logging being sent to /var/log/traffic_ops_traffic.log.
Anybody else seeing the same problem?

No; we're running traffic_ops-1.1.3-2001 from the website and have current timestamps in that log.  I would verify permissions on the directory structure to ensure that the trafops user can write to the file(s) and can access the log directory.  Also try taking a look at files opened by the trafops user via lsof or similar.

did you look in /var/log/traffic_ops/ ?

oh, doh. that's what you are saying... i remember we moved it, but too /var/log/traffic_ops/, i thought.

The log has been there before the 1.1.3 update. It seems like this is still
the location but nothing gets written to it anymore
On Fri, Jul 31, 2015 at 11:51 AM, knutsel<EMAIL_ADDRESS>wrote:

oh, doh. that's what you are saying... i remember we moved it, but too
/var/log/traffic_ops/, i thought.
—
Reply to this email directly or view it on GitHub
https://github.com/Comcast/traffic_control/issues/364#issuecomment-126731089
.

Here's what I've done just now.. Nothing gets written to it.
mv traffic_ops.log ~
#service traffic_ops restart
ls -l /var/log/traffic_ops/traffic_ops.log
-rw-r--r-- 1 root root 0 Jul 31 15:58 /var/log/traffic_ops/traffic_ops.log
[root@cdn1cdcms01 traffic_ops]# rpm -qa | grep traf
traffic_ops-1.1.3-2001.x86_64
On Fri, Jul 31, 2015 at 12:34 PM, Steve Malenfant<EMAIL_ADDRESS>wrote:

The log has been there before the 1.1.3 update. It seems like this is
still the location but nothing gets written to it anymore
On Fri, Jul 31, 2015 at 11:51 AM, knutsel<EMAIL_ADDRESS>wrote:

oh, doh. that's what you are saying... i remember we moved it, but too
/var/log/traffic_ops/, i thought.
—
Reply to this email directly or view it on GitHub
https://github.com/Comcast/traffic_control/issues/364#issuecomment-126731089
.

Did you check whether the file was open via lsof?  I just removed both access.log and traffic_ops.log and they were recreated with the ownership of root:root, which matches what you saw.  The access.log file is being written to, however, I don't have any messages in traffic_ops.log.  That being said, this may be normal, as we have been working to reduce the cruft that appears in the application log.  If that's the case, try increasing the logging from ERROR to DEBUG in /opt/traffic_ops/app/conf/production/log4perl.conf, then restart traffic_ops.  You should then see debug output in traffic_ops.log.

Jeff, I'm not getting DEBUG messages. Thanks!
On Fri, Jul 31, 2015 at 1:15 PM, Jeff Elsloo<EMAIL_ADDRESS>wrote:

Did you check whether the file was open via lsof? I just removed both
access.log and traffic_ops.log and they were recreated with the ownership
of root:root, which matches what you saw. The access.log file is being
written to, however, I don't have any messages in traffic_ops.log. That
being said, this may be normal, as we have been working to reduce the cruft
that appears in the application log. If that's the case, try increasing the
logging from ERROR to DEBUG in
/opt/traffic_ops/app/conf/production/log4perl.conf, then restart
traffic_ops. You should then see debug output in traffic_ops.log.
—
Reply to this email directly or view it on GitHub
https://github.com/Comcast/traffic_control/issues/364#issuecomment-126755803
.

We need a few things before we can proceed:

Run an lsof and grep for "/var/log/traffic_ops"
Run an lsof and grep for log4perl.conf
The contents of the log4perl.conf file found in the previous step

Based on that we'll have a better idea where to go next.

I'm sorry... That should have read "I'm getting debug messages".
On Friday, July 31, 2015, Jeff Elsloo<EMAIL_ADDRESS>wrote:

We need a few things before we can proceed:

Run an lsof and grep for "/var/log/traffic_ops"
Run an lsof and grep for log4perl.conf
The contents of the log4perl.conf file found in the previous step

Based on that we'll have a better idea where to go next.
—
Reply to this email directly or view it on GitHub
https://github.com/Comcast/traffic_control/issues/364#issuecomment-126782927
.

Phew! Looks like this is a non-issue and our cleanup was actually making things more quiet. I'm going to close this issue for the time being, but please let us know if we need to reopen it.

We could probably place a "traffic Ops {version} is starting" is the log.
Just so nobody complains the logs aren't working :)
On Friday, July 31, 2015, Jeff Elsloo<EMAIL_ADDRESS>wrote:

Closed #364 https://github.com/Comcast/traffic_control/issues/364.
—
Reply to this email directly or view it on GitHub
https://github.com/Comcast/traffic_control/issues/364#event-370868947.

Fair enough, that's a good point.  I'm going to reopen this and tag it as a new feature request to ensure it gets done.


span_fn should add tag error=true for functions returning Result when is_err is true

It would be nice to automatically get error tags, which jaeger and other frontends have special ui treatment for.

You can assign this to me, it'd be nice to have for some stuff we're working on


how to change the font of  second p

Challenge Change the Font Size of an Element has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0.
Please describe how to reproduce this issue, and include links to screenshots if possible.
 Browser and Text zoom should be at 100%.
Please use the Help chat room for getting challenge related help. Happy Coding!


Add a cleaner way to make package wrappers

The current way to make packages wrappers is to use a *-with-plugins package.
The original *-with-plugin package, pidgin-with-plugins is mentioning that part of the code could be avoided if buildEnv created the directories.
It feels like the wrapper code could be factored out in a cleaner stdenv.mkWrapper function.
List of packages currently using this technique:

./pkgs/applications/audio/deadbeef/wrapper.nix
./pkgs/applications/audio/puredata/wrapper.nix
./pkgs/applications/networking/instant-messengers/pidgin/wrapper.nix
./pkgs/tools/inputmethods/fcitx/wrapper.nix
./pkgs/tools/inputmethods/ibus/wrapper.nix
./pkgs/applications/misc/rxvt_unicode/wrapper.nix

Wrappers also raise lint issues (cf #11254 ) that could be fixed by inheriting some of the base package meta attributes:
22:06:18.262 > ibus-with-plugins: Lacks a maintainer
22:06:18.263 > ibus-with-plugins: Lacks a platform
22:06:18.263 > ibus-with-plugins: Lacks a license
22:06:18.263 > ibus-with-plugins: Lacks a description

cc pidgin-with-plugins creator @abbradar

Truth be told, I've thought so too -- but to make it properly, we first need to get https://github.com/NixOS/nixpkgs/issues/4621 addressed -- I think we have even more such wrappers now (for more examples see e.g. games/dwarf-fortress/dwarf-therapist/wrapper.nix).

There is also vim_configurable and similar wrappers that try to bundle a config to an executable.

Fixed by  #15002


How are volumes being used?

Hey,
awesome repo!
I have a question. Why there are volumes exist in your setup. Are they being used by the apps or will be the ssd disk used of esch worker?
additionaly why does every worker has a volume since I dont think they are mirrored (like a raid).
im asking this because im not sure how to setup a backup here. Doing it with coolifies s3 backup with hetzner s3 storage, aws s3 or volumes.
Any recommendations here?
regards

@regenrek Sorry for the late response - I didn't see or expect an open issue ticket. :) Currently, volumes are automatically attached to /mnt/hetznerxxxxx, and configuration needs to be done manually. How you use them is up to you. I plan to automate this task with Ansible. You can remove the volumes in your setup if the node storage is sufficient for your needs.
To be honest, I haven't used backup inside Coolify, but using an existing AWS bucket is a good choice... or you can self-host MinIO on a single-node K3s and deploy MinIO operator and tenant with Helm charts

Volumes are removed from main.tf, they can be readded according to the module's docs.
https://github.com/Ujstor/terraform-hetzner-modules


kf_lidar_track.cpp

I want to know if  the Lidar based KF algorithms  have been implemented  ,just can't found the core codes. @kitsukawa

@amc-nu Can you help him?

Not yet published. We are trying to complete it now.
2016/12/14 午後11:51 "Yuki Kitsukawa"<EMAIL_ADDRESS>
@amc-nu https://github.com/amc-nu Can you help him?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/CPFL/Autoware/issues/544#issuecomment-267053363, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAToC2Iz85ty66LiBnpi1z9A7QkpyZQKks5rIAKOgaJpZM4LMkTv
.

kf_lidar_track was implemented at feature/lidar_segmentation branch now.
we will soon merge it into develop.


Compress-477 : add zip64 support for split zip

As it's said in #86

While rebasing I realized that master says the disk number start is a long while this branch uses an int. Actually both master and this branch neglect the possibility of the number of disks requiring Zip64 extra handling. TBH I find the idea of a split ZIP archive that spans more than 64k files a bit disturbing and would address it with a separate change independent of this PR.

This PR is about the zip64 for split zip and some bug fixing in existing code.

For Central File Header
As it's written in Zip Specification :

4.4.13 disk number start: (2 bytes)
The number of the disk on which this file begins.  If an
archive is in ZIP64 format and the value in this field is
0xFFFF, the size will be in the corresponding 4 byte zip64
extended information extra field.

Which means the disk number start may also exceed the maximum value 0xFFFF and may exist in extra field. The existing code in createCentralFileHeader ignored this possibility and didn't detect it.
That's why I added || ze.getDiskNumberStart() >= ZIP64_MAGIC_SHORT for this.
And some testcases in Zip64SupportIT are modified correspodingly :
(1) extra field length : 28 -> 32
(2) size of extra : 24 -> 28

For End Of Central Directory :
There are 6 variables that may exist in Zip64 End Of Central Directory and need to be checked :

numberOfThisDisk
cdDiskNumberStart
numOfEntriesOnThisDisk
numberOfEntries/entries.size()
cdLength
cdOffset

The existing code only checked the numberOfEntries/entries.size() and cdOffset. So the check for the other 4 variables are added in this PR.

Some testcases for zip64 split zip are added(which creates 70,000+ split segments).

When I was writing the testcases, I found it's not easy to test the Zip64 exceptions in End Of Central Directory - the exceptions are always thrown when writing the Central Directory Header. So the newly added exceptions are not tested - I didn't find a proper way to test them.
P.S :
When I was trying to extract the 70,000+ split segments using ZipSplitReadOnlySeekableByteChannel, I got an error : too many open files. I realized that the ZipSplitReadOnlySeekableByteChannel and MultiReadOnlySeekableByteChannel may open all the split segments before reading them - and I think this is meaningless and could be improved. I'm trying to improve them using a 'open files when needed' way. Therefore we don't need to open all the files before reading, which may use a lot of file handles of OS, and we can extract split zips that contain a lot of split segments.

Coverage decreased (-0.04%) to 87.032% when pulling 27fd2033d1218a652f09f59aa3d52db2b4d839ab on PeterAlfreadLee:COMPRESS-477-zip64 into f16125435723d143183c9c45fd3a0f92bdf97ba3 on apache:master.

When I was trying to extract the 70,000+ split segments using ZipSplitReadOnlySeekableByteChannel, I got an error : too many open files. I realized that the ZipSplitReadOnlySeekableByteChannel and MultiReadOnlySeekableByteChannel may open all the split segments before reading them - and I think this is meaningless and could be improved. I'm trying to improve them using a 'open files when needed' way. Therefore we don't need to open all the files before reading, which may use a lot of file handles of OS, and we can extract split zips that contain a lot of split segments.

I have finished this and have tested it(even through it's pretty time consuming when there're a lot split segments).
I'm not sure if this(open file when it's needed to be read) is needed. What do you think? @bodewig
If this is needed, I'm not sure if I should push another seprate PR when this PR is merged(as zip64 split is needed to test it), or I can just push the code in this PR?

This is not really a new issue introduced with the split zips feature, MultiReadOnlySeekableByteChannel has been keeping channels open before as well.
Honestly I haven't got any idea of how much "jumping around" reading split archives (zip or 7z) actually involves. In both cases we read the channels containing file meta data once and will likely never go back to them. So a small number of open channels may be sufficient. OTOH I expect most split archives to only have a pretty small number of parts, an archive with more than 100 parts is something that I don't expect to be common. Therefore making sure we don't open too many streams is probably not that important at all.
I realize I'm not answering your question :-)
I'd make the number of simultaneously opened streams configurable. "infinite" might even be a good default so that people only need to deal with it explicitly if they run into trouble. But in real life situations I'd expect any number bigger than say 20 to have the same effect as infinity (i.e. all channels are open as there are no more than 20 channels anyway).

This is not really a new issue introduced with the split zips feature, MultiReadOnlySeekableByteChannel has been keeping channels open before as well.

Yes, you're right. This is not a new issue with split zips. And as we all know that MultiReadOnlySeekableByteChannel is also used in 7zip split segments, I believe it's also affected by this.

OTOH I expect most split archives to only have a pretty small number of parts, an archive with more than 100 parts is something that I don't expect to be common. Therefore making sure we don't open too many streams is probably not that important at all.

I agree. Actually I never meet any split zips with more than 20 segments. This is nothing common but only a special use case that's introduced in the zip specification.
I agree that it's a extremely special use case and may not be used in most cases. Actually I have already finished this. So I will push the PR and it's up to you to decide whether to merge it or not. @bodewig


Update dependencies

I'd like to get the fix added in https://github.com/micromark/micromark/pull/158.
I've tested this against my local project and it seems to work fine.
There are a few minor breaking changes, aside from all now requiring Node 16.

<EMAIL_ADDRESS>entities → characterReferences
<EMAIL_ADDRESS>some changes around code fences
<EMAIL_ADDRESS>

Hi @florianeckerstorfer, I’m wondering if you’re still active on this project?

@Porges Yes, but I just came back from vacation. I will take a look on the weekend. It might be time to retire Node 12.x support

@florianeckerstorfer thanks!


Feature Request:  Fader Levels for sends to Bus 1-12

Hi there!
Our Online Livestream is run via Stream Deck.
We would love to see the Channel Fader Level that is being sent to a Bus.
Is it possible that we could get a new set of variables?

I'm going to need to look into this but I don't think this comes back with the standard state update, and this would be A LOT of calls to get this right. I think its 48 channels that can be sent to the 16 buses so that's 768 calls that would need to be done every however often which will make Companion very slow and possibly even cause chaos with X32 especially with things like fading
MIGHT be able to do it with... I think It was \batchsubscribe or something like that but as far as I recall I wasnt able to get that to work as expected without causing lag when I tried last time

It should only need to request all the values when connecting to the x32, after that any updates will be pushed to us because we subscribe to state changes with /xremote.
But yeah, making 768 more requests for data is not going to be great for performance when we do connect.
Reading those unofficial OSC docs, I don't think /batchsubscribe is what we need, but perhaps /node is what we need to be able to bulk load data? of course that does require more effort in loading the data and might result in a want to rethink how the known state is stored internally

@Julusian, will /xremote keep those new values up to date? I stand corrected but doesnt /xremote only bring certain fields data values back?
It might be /node that I'm thinking of (apologies it has been a while was quite fun trying to remember how to get my dev environment up and running earlier 😂 next lets see if I remember how to speak TypeScript and how often I use the wrong language) at this point I cannot even remember where we would need to add the /node for initial values. Its going to be an interesting weekend of rediscovery

yeah Im pretty sure that /xremote tells us of all changes made on the x32 other than metering.
So using /node to do the initial load, then leaving the rest to /xremote I think will work.
It has been far too long since I've done anything to this module, so I too have no idea of how it works :)


added website code for containerd.io

Signed-off-by: Patrick Chanezon<EMAIL_ADDRESS>

Looks like @chanezon is going to be a developer again.. good :joy:

@chanezon could you remove some of those background images that are not used before we commit into the tree.  I believe we only use one of the images.

@estesp :-)
@crosbymichael done

LGTM

LGTM


Investigate backend structure

We need to investigate a database/backend structure to be used in Jupiter. Currently, it's using demo data provided in a json file, though this approach is only appropriate for prototyping.

@WillieCubed Do you have any recommendations for a backend solution to begin prototyping with?

For now, exposing this data through custom React hooks is fine. We'll be using Google Cloud resources and likely MongoDB for a database system. For authentication, we'll start with Firebase Auth.

As it stands, I am in full support of using mongo-db, I installed mongo-db tools and will work on making a mock back-end where you could get and post organizations and users.
Also, firebase auth makes the most sense to me too!

@WillieCubed waiting for access to infrastructure

@Buchimi Are we still waiting on access for this issue?

I have been thinking about using gin, this go backend but right now I'm scared of how the developer experience will be like as I have not tried it out yet

Is this what you're referring to? https://github.com/gin-gonic/gin

yes but moved on to django for python

Do you have a branch that I can look at and give some input on so that we can start making plugging those parts into the site?

Here https://github.com/Buchimi/Jupiter-backend-prototype

I did some tests with forms and got some errors that Django didn't support https requests in development servers.
Looked through stack overflow but couldn't fix the problem. I may have to shop around for alternatives as I figure out a solution.

@RubenOlano tell me more about it

TRPC Docs
It kinda integrates nicely with nextjs and provides types with typescript so we can integrate it with our frontend. The only issue with this would be having to refactor some code to use it

This looks like a querying tool as opposed to a backend framework. This is more in line with graphQL as opposed to something like django or express

Yeah, the idea would be to use NextJS for each request and then have trpc handle those requests to the database.

Let's talk about this next meeting


Remove unusual public in Groovy example code

The public access modifier is quite unusal in Groovy code.
Groovy treats fields without access modifier as properties and so public is kind of the default.

This is completely true, I missed this PR :sweat_smile: