Any idea how to add checkboard control code for PythonAPI use?

Discussed in https://github.com/carla-simulator/carla/discussions/5519

Originally posted by xiao6768 June 16, 2022
Hi there,
My purpose is to add checkboard into Carla so that 3rd-party open source AD solution Autoware.AI may use checkboard for Lidar && Vision calibration.
Firstly ,there is a similar request https://github.com/carla-simulator/carla/issues/2696  but not complete yet. I have to do by myself.
At this point I had build mesh-components , by putting into Town03 map,  vision and lidar detection as expected shown as below.

Now I want to control them by carla-python-client.  like "SpawnActor(checbboard_blueprint, transform)",
Then I do:

change checkboard component's parent class to CarlaActorBluePrintFactory
Implement "Make Board Denfinition" in *.h file

BTW, my envirement is ubuntu18.04, source code build. UE4 : 4.24.3  carla: 0.9.10
By recompile all (make launch , then play),
Use following code , I EXPECTED checkboard will be appear on blueprint_library.  BUT NOT.
"
import carla
client = carla.Client('<IP_ADDRESS>', 2000)
client.set_timeout(3)
world = client.get_world()
bp_library = world.get_blueprint_library()
print(bp_library)
"
Any suggestion would be great appreciated!!!!
thanks.

Good news! This issue has been resolved . We are planning to merge into main branch, need approve from company.
Any one please feel free to contact me if you have any questions.

I've been looking for it for a long time. This is exactly what I need. Is it convenient to share the checkboard map or code? Thank you very much

Hello everyone,
As part of our recent research work, we needed to have exact (calibrated) values for all CARLA cameras' intrinsic parameters (RGB, Depth, DVS). Accordingly, we added calibration targets (Checkerboard and AprilGrid) to the Town3 map, and calibrated RGB, and DVS cameras using SOTA algorithms. Check out the repo here: IBISCape.
Thanks.

‘Then, open the CARLA-Unreal Engine interface and press Compile, Build and finally Play.’
I'm really sorry, my Carla is installed under ubuntu20, and I can't find where to open the CARLA-Unreal Engine interface，I tried to replace the map file directly, but failed
I want to know how to import the map files

Firstly, My compile Env. is ubuntu18.04, carla 0.9.10, and UE4 is 4.24.3.
You have to following this instruction get UE4 source code compile OK, then compile Carla source code.


~/go-i2p and ~/go-i2p/config are not created automatically

This is on the noise experimental branch:
require (
	github.com/go-i2p/go-i2p v0.0.0-20241023173406-c149eef1deaf
)

main:
import (
	"github.com/go-i2p/go-i2p/lib/router"
)
func main() {
	r, err := router.CreateRouter()
	if err != nil {
		panic(err)
	}
	r.Start()

	r.Stop()
}

Found an error when starting a new router. The cause being that the directories required aren't being created automatically.
rayquaza@workstation:/mnt/FIVEHUNDREDSSD/enc/Programming/go/go-i2p_main/bin$ ./go-i2p_main
DEBU[0000] Logging enabled.                              fields.level=debug
DEBU[0000] Creating router with default configuration
DEBU[0000] Creating router from configuration            config="&{/home/rayquaza/go-i2p/base /home/rayquaza/go-i2p/config 0x62ba60 0x626a00}"
DEBU[0000] Router created successfully from configuration
DEBU[0000] Router created successfully with default configuration
DEBU[0000] Starting router
DEBU[0000] Stopping router
DEBU[0000] Entering router mainloop
INFO[0000] Creating new StdNetDB                         db_path=/home/rayquaza/go-i2p/config/netDb
DEBU[0000] Created StdNetDB                              netdb_path=/home/rayquaza/go-i2p/config/netDb
DEBU[0000] NetDB directory does not exist, creating it
DEBU[0000] Creating network database directory           path=/home/rayquaza/go-i2p/config/netDb
ERRO[0000] Failed to create root network database directory  error="mkdir /home/rayquaza/go-i2p/config/netDb: no such file or directory"
ERRO[0000] Failed to ensure NetDB                        error="mkdir /home/rayquaza/go-i2p/config/netDb: no such file or directory"
DEBU[0000] Calculating NetDB size
DEBU[0000] Recalculating NetDB size
DEBU[0000] Recalculating NetDB size
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4ef14d]

goroutine 7 [running]:
github.com/go-i2p/go-i2p/lib/netdb.(*StdNetDB).RecalculateSize.func1({0xc00001e1e0, 0x22}, {0x0?, 0x0?}, {0x56f7e0, 0xc0000bad80})
	/usr/local/go/bin/pkg/mod/github.com/go-i2p/go-i2p@v0.0.0-20241023173406-c149eef1deaf/lib/netdb/std.go:142 +0x4d
path/filepath.Walk({0xc00001e1e0, 0x22}, 0xc0000d3b38)
	/usr/local/go/bin/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.1.linux-amd64/src/path/filepath/path.go:425 +0x4a
github.com/go-i2p/go-i2p/lib/netdb.(*StdNetDB).RecalculateSize(0xc00007e108)
	/usr/local/go/bin/pkg/mod/github.com/go-i2p/go-i2p@v0.0.0-20241023173406-c149eef1deaf/lib/netdb/std.go:141 +0xae
github.com/go-i2p/go-i2p/lib/netdb.(*StdNetDB).Size(0xc00007e108)
	/usr/local/go/bin/pkg/mod/github.com/go-i2p/go-i2p@v0.0.0-20241023173406-c149eef1deaf/lib/netdb/std.go:99 +0x1b7
github.com/go-i2p/go-i2p/lib/router.(*Router).mainloop(0xc00007e100)
	/usr/local/go/bin/pkg/mod/github.com/go-i2p/go-i2p@v0.0.0-20241023173406-c149eef1deaf/lib/router/router.go:92 +0x1e5
created by github.com/go-i2p/go-i2p/lib/router.(*Router).Start in goroutine 1
	/usr/local/go/bin/pkg/mod/github.com/go-i2p/go-i2p@v0.0.0-20241023173406-c149eef1deaf/lib/router/router.go:78 +0x1d3

I also had to manually create ~/go-i2p and ~/go-i2p/config in order for it to work
rayquaza@workstation:/mnt/FIVEHUNDREDSSD/enc/Programming/go/go-i2p_main/bin$ mkdir ~/go-i2p/config
rayquaza@workstation:/mnt/FIVEHUNDREDSSD/enc/Programming/go/go-i2p_main/bin$ ./go-i2p_main
DEBU[0000] Logging enabled.                              fields.level=debug
DEBU[0000] Creating router with default configuration
DEBU[0000] Creating router from configuration            config="&{/home/rayquaza/go-i2p/base /home/rayquaza/go-i2p/config 0x62ba60 0x626a00}"
DEBU[0000] Router created successfully from configuration
DEBU[0000] Router created successfully with default configuration
DEBU[0000] Starting router
DEBU[0000] Stopping router
DEBU[0000] Entering router mainloop
INFO[0000] Creating new StdNetDB                         db_path=/home/rayquaza/go-i2p/config/netDb
DEBU[0000] Created StdNetDB                              netdb_path=/home/rayquaza/go-i2p/config/netDb
DEBU[0000] NetDB directory does not exist, creating it
DEBU[0000] Creating network database directory           path=/home/rayquaza/go-i2p/config/netDb
DEBU[0000] Calculating NetDB size
DEBU[0000] Recalculating NetDB size
DEBU[0000] Recalculating NetDB size
INFO[0000] Finished recalculating NetDB size             count=0
DEBU[0000] Updated NetDB size cache file
DEBU[0000] NetDB Size                                    size=0
DEBU[0000] Router ready                                  at="(Router) mainloop"
^C

proposed fixed in https://github.com/go-i2p/go-i2p/pull/17


Add thrust output iterator fix to rapids-cmake thrust patches

Description
Backports  https://github.com/NVIDIA/thrust/pull/1805 to thrust 1.17.2 since it looks to be slated for thrust 2.1 . Adding the patch to ensure that all RAPIDS projects won't be hit by this issue
Checklist

[x] I am familiar with the Contributing Guidelines.
[x] New or existing tests cover these changes.

@gpucibot merge


Expand Expr integer type-width inference to all binary operations

Summary
This expands the inference on integer widths that was previously only done in the bitwise operations to all binary operations, to reduce the number of cast nodes that will be inserted.  This is mostly a quality-of-life improvement (for us as well), because there's not really a need to include casts when we could just have our integer literals start off at the correct size.
In the immediate term, this is useful for OpenQASM 3 export for the IBM QSS stack, which doesn't support explicit casting yet.

Details and comments
This turned up during my testing of our capabilities against the QSS stack.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

11 of 11 (100.0%)  changed or added relevant lines in 1 file are covered.
39 unchanged lines in 6 files lost coverage.
Overall coverage decreased (-0.04%) to 86.048%

Files with Coverage Reduction
New Missed Lines
%

crates/qasm2/src/expr.rs
1
93.76%

qiskit/circuit/tools/pi_check.py
1
91.23%

qiskit/pulse/library/waveform.py
3
93.75%

qiskit/circuit/classical/expr/constructors.py
4
91.14%

crates/qasm2/src/lex.rs
6
91.14%

crates/qasm2/src/parse.rs
24
96.18%

Totals

Change from base Build<PHONE_NUMBER>:
-0.04%

Covered Lines:
72601

Relevant Lines:
84373

💛  - Coveralls


Dowload-dependencies.sh gives a vague error when URL not found

I get this on Ubuntu 19.10:
$ ./download-dependencies.sh
curl: (22) The requested URL returned error: 404 Not Found

I don't know which URL it doesn't find. Is it possible to give a more descriptive error message with the URL that isn't found?

Adding -v to the curl command in the function maybe_download shows that it's about this URL:
https://github.com/synesthesiam/rhasspy/releases/download/v2.0/rhasspy-tools_amd64.tar.gz

Hi,
I spent my whole day getting out of this.
I modify the download-dependency.sh
`# -----------------------------------------------------------------------------
Rhasspy
-----------------------------------------------------------------------------
echo "starting Rhasspy downloads"
for FRIENDLY_ARCH in "${FRIENDLY_ARCHS[@]}"; do
rhasspy_files=("rhasspy-server_2.4.10_amd64.deb")
for rhasspy_file_name in "${rhasspy_files[@]}"; do
rhasspy_file="${download_dir}/${rhasspy_file_name}"
rhasspy_file_url="https://github.com/synesthesiam/rhasspy/releases/download/v2.4/${rhasspy_$
echo rhasspy_file_url
echo "https://github.com/synesthesiam/rhasspy/releases/download/v2.4/${rhasspy_file_name}"
maybe_download "${rhasspy_file_url}" "${rhasspy_file}"
done
done`
Then you have to modify the create-venv.sh cause some dependencies are broken.
I was forced to add --system-site-packages to the venv creation in order to install all packages "by hand" in my server (like openfst that can't be install easily from within the virtual env).
And lastly i modify the following part to install the .deb.
# Extract Rhasspy tools rhasspy_tools_file="${download_dir}/rhasspy-server_2.4.10_amd64.deb" echo "Extracting tools (${rhasspy_tools_file})" dpkg -i "${rhasspy_tools_file}" #-C "${venv}" -xf "${rhasspy_tools_file}"
Hopping that will help,
76MPaul

I messed up and re-named the amd64 to arm64 🤦♂  I've renamed it back, so hopefully the download script will work again. Sorry!
Depending on whose asking, it seems arm64 is either called that, aarch64, or arm64v8. Different tools seems to require different names, so I'm failing to keep it consistent across my build pipeline. Too many manual steps.

It's working now.


V3 Proportional Adds Issue

Investigate issue mentioned by Alberto.

Initial thought is this could be similar to CowAmm issues and we need to replicate maths 100%. Currently there may still be changes to maths (and between deploy versions) so would wait until that is final.

From Alberto:
The v3 proportional adds and removes are merged and working as expected but I found one issue in the permit2 signature step:
I hit this AMOUNT_OUT_OF_RANGE error when setting the amountsIn to MaxAllowanceTransferAmount (MaxUint160)  in the Permit2Helper.signAddLiquidityApproval call.
I didn’t debug the SDK code but saw that I was setting:
MaxUint160 -><PHONE_NUMBER>330902918203684832716283019655932542975n
but the number that you check here is slightly bigger:
1462963138968233821121888517548999302675588475517n (edited)
Given that the proportional amount do not have slippage, I guess that It could be the fees being added? (I can check the code if you don’t have the answer)
If that’s the case, we should detect that the user is allowing the maximum MaxUint160 and avoid adding the fee.
For now. I’m just using MaxUint159 (which works like max in practice) so no hurries at all

Are we still expecting changes to maths or is it ok to work on this issue?

Are we still expecting changes to maths or is it ok to work on this issue?

Juani said yesterday that the current testnet deployment should be final maths.

Ok, I see what's happening here.
The PermitHelper was designed to approve only the user amountsIn + slippage with safety in mind, in the sense that it's safer to approve only what's going to be used instead of the MaxAllowance of MaxUint160.
While the FE opted for convenience, in the sense that user will approve MaxAllowance only once.
I can apply the suggested approach to make sure we never set anything higher than MaxUint160 by mistake, but maybe a better solution is to adapt the PermitHelper to match FE logic.
Since it's an opinionated helper to suit our FE needs, I don't see a reason to have this logic in the SDK (matching user input amounts) that will be overwritten by the FE.
What do you guys think?
cc @agualis @johngrantuk

Both options are straightforward to implement for us. I don't thing that other integrators will need those SDK checks so I think it makes sense to remove them until someone really needs them for an alternative ux approach

Bruno 19 Nov 20024:
Talking to John we believe that it makes more sense to approve only the amountsIn + slippage within the Permit2 signature.
For that, the FE can simply forward the add liquidity input the user provided.
Any reason why the FE adapted it to max allowance?
Let me know if you guys are ok on making that change or if you’d still rather maximize the allowance.
Thanks!
alberto:
Ok, so our approach for v3 permit2 within V3 was:
the user approves MaxAllowance with 24 hours of expiration
within the next 24 hours they can add any amount of that token (cause MaxAllowance is, in practice, enough to do as much transactions as they want)
Here you can find the implementation for Swaps:
https://github.com/balancer/frontend-monorepo/blob/main/packages/lib/modules/tokens/approvals/permit2/signPermit2Swap.tsx#L64
bruno:
ok, so if you still want to use MaxUint160, all you should do is to set slippage to zero
alberto
I don’t say that this is better that other possible UX alternatives but this is what we (with Pon and Gareth) decided to test
For example, Uniswap uses exact amount for 30 days if I’m not wrong
But we can always change this if we find good arguments
bruno:
ok, so if you still want to use MaxUint160, all you should do is to set slippage to zero
In practice, this should work and does not require changes on your side
alberto:
So I would go with this solution and we can have a new discussion if we decide to change the UX, sounds like a plan?
bruno:
I agree it’s best to leave the SDK flexible to accept any input amount from the FE - then you can experiment on your end
just keep in mind that slippage will be applied to the amounts provided
I’ll close the ticket then


Use consistent whitespace in template placeholders

Sorry, I know this a trivial PR, but it would be nice to have consistent use of whitespace inside all charts. 😉

Thanks a lot. This is very much appreciated. But I'm afraid most of this, if not all, has already been done in #1062 which has just been merged today.

@unguiculus Hmm, the issue you mentioned is about Ealsticsearch. 🤔

Ouch, sorry. I guess I got confused by the merge conflicts. Here's an open PR for Artifactory which should get merged soon: #1314.

@unguiculus I updated the PR to resolve the merge conflicts. My PR doesn't just cover Artifactory, it fixes the template placeholder whitespace in all charts.

Wow, I guess I did not look far enough. Did you use some script for this?
Usually we want separate PRs per chart. Do you think you can split this up?
@kubernetes/charts-maintainers How should we go about this? I think this is great.

Just a simple Regex search & replace in Sublime Text:
{{([^ -]) -> {{ $1
([^ -])}} -> $1 }}
Splitting it up would have ended up in 38 separate PRs just adding whitespace, so I left it in one single PR instead. 😉
https://github.com/kubernetes/charts/pull/1437/files?w=1 shows that it's really only whitespace that has changed.

Maybe we can take this as a whole. Discussing it on Slack.

/test pull-charts-e2e

Rebased master, fixed tensorflow helper.
/retest

/retest

@unguiculus Sorry, not sure what's wrong with the Jenkins built, I don't think it's actually related to my changes.

Yeah, there seems to be some test flakiness. We are going to merge this as a whole. Anything else does not make sense.

Would you mind covering NOTES.txt files as well? I understand that those will not be as easy because they often contain shell magic. But we can just try and get this in soon and follow up with that in a separate PR.

Sure, I can do that as well, but I'd prefer a separate PR to minimize impact.

@unguiculus Fixing NOTES.txt wasn't so bad, so I added it here. Also fixed parameter order on three occasions, so the command is always identical. Should I also adjust APP_HOST vs. SERVICE_IP? (I'd prefer the latter, it is more precise.)

I'm going to merge this as is. You may create a separate PR as follow-up. I'd prefer SERVICE_IP as well.

Awesome, thanks!

Hi, this commit breaks (at least) the grafana chart. It fails a lint with the default values.
[ERROR] templates/: parse error in "grafana/templates/job.yaml": template: grafana/templates/job.yaml:41: unexpected "}" in operand

Error: 1 chart(s) linted, 1 chart(s) failed

As a follow up, would it be worth having CI lint each of the charts in branches before merging?

@rk295 Sorry, didn't notice the change in the Job.
Fixed in #1458.


Leveled compaction crashes when recovering from manifest

The LeveledCompactionController::apply_compaction_result method tries to sort the merged SSTs by key inside the method, which will cause problem when this method is called in manifest recovery context, because at that point no actual SSTs are loaded.

There is indeed a problem with this,  sorting can be extracted from apply


feat: improved installation mechanism

closing while i write tests to prevent github actions spam

Hey, the previous README mentioned the need to do a second rebase, that doesnt seem to be handled by the new script?

Hey, the previous README mentioned the need to do a second rebase, that doesnt seem to be handled by the new script?

it's handled by a firstrun service


import { useMediaQuery } from "@/hooks/use-media-query" can't be found

I'm trying to follow this example Drawer but when I add to my code
import { useMediaQuery } from "@/hooks/use-media-query"

const isDesktop = useMediaQuery("(min-width: 768px)")

I'm seeing the error that use-media-query can't be found

Hi @NicoCaldo, the use-media-query doesn't come with the Drawer. However you can get it from the source code.
I'll it here just in case:
import * as React from "react"

export function useMediaQuery(query: string) {
  const [value, setValue] = React.useState(false)

  React.useEffect(() => {
    function onChange(event: MediaQueryListEvent) {
      setValue(event.matches)
    }

    const result = matchMedia(query)
    result.addEventListener("change", onChange)
    setValue(result.matches)

    return () => result.removeEventListener("change", onChange)
  }, [query])

  return value
}

You can copy this and save it in the hooks/use-media-query.tsx file.

Thanks

Could this maybe be included in the docs? Would be much better than having to look around in the source code (or in my case, Google) :)

Explanation is missing from https://ui.shadcn.com/docs/components/combobox#responsive as well.


Pass-through authentication is not currently supported when using Windows 10 devices joined to Azure AD.

For PTA it is written Pass-through authentication is not currently supported when using Windows 10 devices joined to Azure AD. Article from01/03/2018.
PTA Current Limitations article (https://docs.microsoft.com/en-us/azure/active-directory/connect/active-directory-aadconnect-pass-through-authentication-current-limitations) says :"Supported Scenarios: Azure AD domain joins for Windows 10 devices". Article from 01/09/2018.
I think there is a contradiction between both articles.
I would be grateful if you could clarify this aspect.
Thanks a lot.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c117162e-2efd-b2bd-5a44-f461a9425e81
Version Independent ID: 29ca7351-9111-0e90-83a0-44e77b94598d
Content
Content Source
Service: azure

@Yazooo  Thank you for the valuable feedback. I forwarded your feedback to the author.

@Adam-Smith-MSFT Please assign this issue to @billmath for resolution.

@Yazooo  thank you for pointing this out.  I need to follow-up with the PM and verify this.  I believe the pta limitations doc is more accurate but let me verify.  Thanks!

@Yazooo  wanted to let you know that I did confirm that it is supported.  I have updated the doc you commented on and the statement about it being unsupported should be gone later today.  Thanks again for this.  Let me know if there is anything else, otherwise I will go ahead and close this later today.
Thanks!

@Yazooo this has been fixed and should be live now.  Thanks again.
#please-close


useContract

Description
Original written down in #36:
This is what I'm thinking of:

The hook will accept a contract's address and its ABI. It will then parse that ABI and return all the available methods with type definitions. I think this is quite ambitious but is doable and has been done before.
For the sake of simplicity, I suggest starting off with two basic methods that the hook will return for all contracts: readContract and writeContract. These methods won't contain type defs for now but will let you interact with the contract directly.

Hey @Dhaiwat10, can I be assigned to this one? Working on a rough draft now. Thanks!

Hey @swellander, are you working on both this one and https://github.com/Developer-DAO/web3-ui/issues/17? I want to make sure both are in progress.

Copy/pasting relevant comment from another issue:
https://github.com/Developer-DAO/web3-ui/issues/36#issuecomment-983508910

Couldn't we use TypeChain to generate the type definitions?

I wanted to +1 on this.

@crondinini yep currently working on this one! It looked like @RoguePenguins was working on #17 so I switched to this issue.
As for TypeChain, could you maybe elaborate on a high level how the integration would work? I've never used the library, but after looking at it a bit I'm still unsure how we would generate types dynamically based off of the abi the user passes to useContract(address, abi). Thanks!

@swellander have you started working on this? Are you blocked in any way? How can I help you?

Hi @Dhaiwat10, thanks for offering to help. I'll share a bit of what I've been thinking so far and maybe you can let me know if I'm on the right track.
Context
The current version of the useContract hook that I have right now looks like this:
function useContract(address: string, abi) {
    const context = React.useContext(Web3Context);
    const [contract, setContract] = React.useState({});
    React.useEffect(() => {
        if (context) {
          const newContract = new Contract(
            address,
            abi,
            context.signer
          )
          const contractInterface = Object.values(newContract.interface.functions).reduce((accumulator, funcFragment) => {
            return {...accumulator, [funcFragment.name]: newContract[funcFragment.name]}
          }, {})
          setContract(contractInterface);
        }
    }, [context])
    return contract;
}

The hook returns contractInterface which is intended to be an object with all of the contract's available methods, accessed through an ethers.js Contract instance.
Blocking Questions

I'm confused about the readContract and writeContract methods you mentioned in this issue's description. Are these like proxy methods that would accept the name of a property or method on the contract?
I'm struggling to come up with a way to type the contract passed to useContract(). The Typechain library mentioned by @christiananese and @crondinini seems like a promising solution, but I haven't been able to think of an effective way to integrate it (see my previous comment on this issue).

Thanks again for your help!
Also, let me know if you think it would be more appropriate to have this discussion in a draft PR instead.

@swellander that's a great starting point, thanks for writing this up. Re. your questions:

Are these like proxy methods that would accept the name of a property or method on the contract?

Yes. But now that I think about it, it's better to just do something like this for now:
const contract = useContract({ address: '0xxxxxababa', abi: '[]' })

// read
contract.greeting();

// write
contract.setGreeting('New greeting');

// payable
contract.transferTo('0x0x0x0x0',  { value: '1000000000000000000' }) // value amount in wei

Basically the same as calling the contract using ethers. I don't think making a distinction between read and write is worth it right now.

Gotcha. How about you create a draft PR and we continue the discussion there? Type definitions are not a priority right now, FWIW. We can add them through a separate PR as they will also change the hooks' interface. :)

@Dhaiwat10 got it. Thanks for the reply!


Filter.By with many-to-many relationship

Hi David, really like this library, but I have an issue with filtering in a many-to-many EF core setup.
I have some models like:
class SecureObject {
  IEnumerable<Membership> Membership { get; set; }
}

class Membership {
  SecureObject SecureObject { get; set; }
  User User { get; set; }
  MembershipType MembershipType { get; set; }
}

class User {
  string DisplayName { get; set; }
}

I'm trying to dynamically filter a db query of SecureObjects by user's display name (among other things).
Is there any way to pass in this query? Can't get it to work right now, but this is what I tried:
filter.By(Membership.User.DisplayName, ...) resulting in:

System.ArgumentException: ''User' is not a member of type 'System.Collections.Generic.ICollection`1[....Model.Membership]''

Any help would be greatly appreciated!

We only filter 2 levels deep and the syntax there is  filter.By("ArrayName[PropertyOfArrayObjectName]", ...);. So in your case it might be "Membership[User][DisplayName]"?


Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll'...

The specified module could not be found.
Running in latest Service Fabric.
Using Microsoft.Azure.DocumentDB.Core 1.4.0
It looks like this was an issue with the none core version of the package as well.

Further clarification. I added the Nuget package to the Service Fabric executable project, hoping that it would cause the ServiceInterop file to be included in the deployment package, but I still got the exception.

We are currently investigating the root cause. If anyone is running into similar issue, you can get unblocked by directly referencing the missing assemblies. We will update this thread once we find the root cause.

This issue can be reproduced in below environment

VS 2017 (15.2) version
.net standard library
.net framework run-time

This is build issue and is covered at https://github.com/dotnet/sdk/issues/901. It also suggests a workaround https://github.com/dotnet/sdk/issues/901#issuecomment-316191996 which addresses the issue.

@kirankumarkolli
If it's not possible for the DocumentDb team to provide a fix for this in the next release (as this seems like a lower level issue) it would be very helpful if a readme was provided in the Nuget package documenting the issue. Even better might be a warning in the Nuget package description.
I do have a related question though. The current Nuget package includes native binaries for windows, does that mean this library couldn't be used on other platforms?

@jackbond we are exploring ways to document known issues and as part of it we will include it as well.
Thanks for feedback.

@jackbond closing this issue.

@kirankumarkolli
I've recreated this problem NOT using Service Fabric and using PackageReferences (as opposed to packages.config) Can you please reopen this issue? Here's a minimal solution that reproduces the problem.
repro.zip

Reopening it to investigate the issue.

@kirankumarkolli
Have you had any luck resolving this issue? This bug is preventing us from using the .netstandard version of the DocDb library, which in turn prevents us from migrating / converting some of our projects to either .netstandard or .netcore.

I had this issue with my Azure Function as well. Solved by uninstalling all the NuGet packages and re-installing the latest Microsoft.NET.Sdk.Functions and Microsoft.Azure.Graphs (which depended on Microsoft.Azure.DocumentDB). A tutorial I was following recommended installing Mono.CSharp. However, by not including this package, my function ran successfully both on my local machine and in Azure.
Hope this helps pin down the issue for someone.

It happens for me as well.

Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

Unable to load DLL 'Microsoft.Azure.Documents.ServiceInterop.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

@loneshark99 what is the project type (Framework, Core, Standard)? Also is SDK nuget installed in the project or in the reference project?

@kirankumarkolli  Sorry it was my issue, I added the .core nuget instead of the client one. Its gone now.


Margin Trading interface

@creed-victor
trading v1.3.xd.zip

Best to have a quick 5 min call before you start on this @creed-victor

@creed-victor I just noticed that the cta's on an open trade have the wrong font. It is work sans in the file. Please use Montserrat, apologies for this mistake

trading v1.4.zip
this is updated after speaking with ororo
there are also some very slight design changes

https://xd.adobe.com/view/f742b57d-92fc-44b4-4f93-49ade24a3795-8173/
[Margin Trade V2.0.xd.zip](https://githu@soulBit stributedCollective/Sovryn-frontend/files/6414149/Margin.Trade.V2.0.xd.zip)
@soulBit these should address the users feedback and have a little tidy up

rUSDT rename done in https://github.com/DistributedCollective/Sovryn-frontend/pull/1025


Modify protocol

I need to change the communication protocol of the Android App "app-hidden-release" in order to get it recognised by the web platform I use.
Can you please tell me which file I should change and how to do that?
My protocol.
!1,gsn;	(gsn:IMEI, the server will establish a connection according to IMEI)
Example: !1,355832050061016;
!D,date,time,lat,lot,speed,degrees,flag,altitude,bat,sta1,sta2,0;
Example: !D,3/7/15,11:39:30,45.589621,11.544391,0.0,225.8,1f0001,12.11,98,0,0,0;
I just need to get even the date is the coordinates...
Thanks,
Best regards,
Marianna

I think you can make changes in app/src/main/java/org/traccar/client/ProtocolFormatter.java


Proxy through server

Describe the feature or improvement you're requesting
I want to meter users by proxying requests through my server, but I still want to use your nice interface to the API instead of creating a new one for the server interaction. So I'd love to be able to send the requests to a custom hostname or base URL, without needing an API key and then I can ship this to iOS apps too.
Additional context
No response

So more or less a way of changing the base URL used for the API? I can add in a parameter to change out the URL for proxies, as long as your own proxy has the same mechanisms of handling traffic from and to the app.

Thanks for the reply. I've read through more of your source and think the ideal for my use case at least would be to be able to swap in both a custom base URL as well as a custom "base API client" so that I can use my own server's auth schemes.
In my case I generate OpenAPI clients for my server via openapi-generator using URLTask async/await in swift5. I'm thinking how best to abstract OpenAI.swift in this project to allow swapping in alternative API clients for users who already have something working for server auth or other network/header needs for delivering the request payload. Maybe define a protocol and let the user swap in something that bridges to the user's API client library

Thanks for the reply. I've read through more of your source and think the ideal for my use case at least would be to be able to swap in both a custom base URL as well as a custom "base API client" so that I can use my own server's auth schemes.
In my case I generate OpenAPI clients for my server via openapi-generator using URLTask async/await in swift5. I'm thinking how best to abstract OpenAI.swift in this project to allow swapping in alternative API clients for users who already have something working for server auth or other network/header needs for delivering the request payload. Maybe define a protocol and let the user swap in something that bridges to the user's API client library to the payload you want to deliver

I'll create a branch for this repo, and I will let you try it out to see if it fits within your requirements. Once a decision is a made, I'll go ahead and release the feature for my next version that'll include Whisper's endpoints.

Thank you, happy to give feedback. For now I've forked to meet my needs but didn't find a nice way to contribute something back upstream, so it's great if you have a better way in mind.

Heyo,
It's a pleasure to have you suggest the feature in the first place. I'd be more than happy to see how your own implementation went. I do agree that having a more advanced feature for proxy servers would be beneficial for this project. Let me know if you're interested in publishing a PR or just willing to share your own code.


[github_actions] Allow manual uploads

This is useful for the scenario when the Fedora Copr API key is expired.

I only modified the *-upload.yml workflows as they are impacted by an expired Fedora Copr API key.
With this simple change, we can run those upload workflows manually: https://docs.github.com/en/actions/using-workflows/manually-running-a-workflow . Automation will still continue to work when PRs are merged into the main branch.

As an update, this ended up being a copy/paste error on my side leading to the new Fedora Copr API key not working. It has since been updated and the playtron-os-files RPM that was affected has now been automatically pushed up to the Fedora Copr.
For preventing this in the future, I have setup automatic Google Calendar reminders for our team internally.
This feature of manual workflow runs could still be useful in the future.


perf._utils.jitwarmupoptions()?

I think that to solve the warmup issue, you may need to add the following options when testing on pypy, e.g.:
pypy --jit threshold=1,function_threshold=1
You can find more details by checking the output of pypy --jit help, by default the JIT will only start working after 1039 loops.
That's why I was suggesting the title of this issue, these command line options would vary by JITted Python implementation, e.g.:

If I understand correctly under Jython under Oracle's JDK, the argument might be jython -J-Xcomp (you'll see that it takes its sweet time starting compared to a regular jython):

-Xcomp
Forces compilation of methods on first invocation. By default, the Client VM (-client) performs 1,000 interpreted method invocations and the Server VM (-server) performs 10,000 interpreted method invocations to gather information for efficient compilation. Specifying the -Xcomp option disables interpreted method invocations to increase compilation performance at the expense of efficiency.

If using Pyston, according to the README at https://github.com/dropbox/pyston, you should try -O (you can see it in action with pyston -O -v):

-O
Force Pyston to always run at the highest compilation tier. This doesn't always produce the fastest running time due to the lack of type recording from lower compilation tiers, but similar to -n can help test the code generator.

If running under IronPython, you probably could try with -X:CompilationThreshold 1

-X:CompilationThreshold  The number of iterations before the interpreter starts compiling

About running with pypy --jit threshold=1,function_threshold=1 this is not a good idea IMHO. Nobody does that in production. It's wasteful. It will cause a lot of tracing and compilation. Probably increase pressure on the garbage collector in some real world benchmarks. Maybe it could win speed in some microbenchmarks. What should be done instead is use default parameter here and warmup the JIT "enough". Not sure how many iterations would be enough though.

It seems like the goal of a benchmarking suite is not necessarily to "win" some arbitrary competition, rather as a point of reference in some ongoing process. So IMO all the tricks to get a better score obscure the real goal. Default setups allow developers to tweak the defaults over time, and compare over a wide range of benchmarks how those play out.
Hopefully any microbenchmark will reach a plateau withing a few seconds of startup, I think measuring that plateau should be the goal. (me mumble something about statistical change detection here)

@squeaky-pl and @mattip explained the rationale: it's not a good idea :-) I close the issue.
I started to document how to benchmark PyPy (JIT compilers): http://perf.readthedocs.io/en/latest/perf.html#jit-compilers


Function 'FlashAttnQKVPackedFuncBackward' returned nan values in its 0th output

I found my training loss become NAN after training many epochs, but the model params were all finite (i.e. torch.isfinite) after the NAN loss occurred (I checked this by loading the saved checkpoint file from disk). I tried to resume the model from the checkpoint file (i.e. the checkpoint after the NAN happened), at the very beginning, the training process just seemed to be okay, but after several epochs, the NAN loss occurred.
I tried to set torch.autograd.set_detect_anomaly(True) and CUDA_LAUNCH_BLOCKING=1 to find out what happened, and the result showed that FlashAttnQKVPackedFuncBackward returned NAN values for its output.
It is very strange, because the model params are all finite, and the NAN happen after several epochs, so the training data should be okay. But if the backward pass produces NAN value, why the model params don't contain any NAN values? BTW, I didn't use any gradient clipping.
I also check all the model params, and they just seem to be fine. The maximum abs max value among all the params (except the non-learnable freqency weight) is 1.415, not too large.
Do you have any suggestions on this? Is the FlashAttnQKVPackedFunc numerically unstable?
Thank you very much! Looking forward to your reply.

Thanks for the report. The function should be numerically stable.
Which commit of FlashAttention are you using? On which GPU? What are the dimensions of the attention?
In order for me to reproduce the issue, can you save the arguments that caused NaN and send it to me? That'd be very helpful.
For example, you can add these lines right before the return statement of the backward function to save the tensors to the file nan_repro.pt:
        if dqkv.isnan().any():
            state_dict = {'dout': dout, 'qkv': qkv, 'out': out, 'softmax_lse': softmax_lse,
                          'cu_seqlens': cu_seqlens, 'max_seqlen': ctx.max_seqlen,
                          'dropout_p': ctx.dropout_p, 'softmax_scale': ctx.softmax_scale,
                          'causal': ctx.causal, 'rng_state': rng_state}
            torch.save('nan_repro.pt', state_dict)
            breakpoint()

Thanks for your help!

I was having the same problem...  Long story short, it looks like dq, dk and dv need to be zeroed-out, since they are used as accumulators?   However, currently, flash_attention_interface allocates them via torch.empty_like.  Setting them to 0 before flash_attn_cuda.bwd seems to have resolved the issue.

I'm very curious about this. I think all the of values in dq, dk, dv should overwritten during the execution of the backward pass.
The only problematic scenario I could imagine is when q, k, v are longer than what cu_seqlens indicate.
For example, if q has shape (10, nheads, headdim) where 10 is supposed to be the total batch * seqlen, then dq is allocated as torch.empty_like(q). If e.g. cu_seqlens = [0, 5, 8] (which says that the batch has 2 sequences, 1st sequence being stored in index 0 -> 4, and 2nd sequence being stored in 5 -> 7), then during the execution only values from indices 0->7 are written, and values from indices 8 -> 9 are not overwritten.
@vadimcn could you say more about your setting? Does it fall into this case?

The only problematic scenario I could imagine is when q, k, v are longer than what cu_seqlens indicate.

Yes, this was indeed the case :facepalm:.    Thanks for the hint!

@tridao I can reproduce this error after a very short time using the script below. Is this a user-side usage error?
import torch
import torch.optim as optim
import torch.nn.functional as F
from functools import partial
from flash_attn.modules.block import Block
from flash_attn.modules.mha import MHA
from flash_attn.modules.mlp import Mlp

embed_dim = 256 # !!! change this to 64 and the error will not be observable !!!
batch_size = 16
num_heads = 8
seq_length = 512
dim_feedforward = 1024
learning_rate = 0.01
device = torch.device("cuda")
torch.set_default_dtype(torch.bfloat16)
torch.autograd.set_detect_anomaly(True)

# Initialize model
model = Block( # TransformerEncoderLayer
        embed_dim,
        mixer_cls=partial(
            MHA,
            num_heads=num_heads,
            use_flash_attn=True,
            rotary_emb_dim=0,
        ),
        mlp_cls=partial(Mlp, hidden_features=dim_feedforward),
        resid_dropout1=0.0,
        resid_dropout2=0.0,
        prenorm=False,
    ).to(device)

optimizer = optim.Adam(model.parameters(), lr=learning_rate)
inputs = torch.full((batch_size, seq_length, embed_dim), 1000.0, device=device)

# Training loop
for i in range(999999):
    print(f'Iteration {i + 1}')
    optimizer.zero_grad()
    output = model(inputs)
    loss = output.mean()
    loss.backward()
    optimizer.step()

Output:
/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py:959: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  def forward(
/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py:1018: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  def backward(ctx, dout, *args):
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
Iteration 10
Iteration 11
Iteration 12
Iteration 13
Iteration 14
Iteration 15
/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/autograd/graph.py:825: UserWarning: Error detected in FlashAttnQKVPackedFuncBackward. Traceback of forward call that caused the error:
  File "/home/otto/Development/temp/test.py", line 41, in <module>
    output = model(inputs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/modules/block.py", line 195, in forward
    mixer_out = self.mixer(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/modules/mha.py", line 670, in forward
    context = self.inner_attn(qkv, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/modules/mha.py", line 122, in forward
    return flash_attn_qkvpacked_func(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 729, in flash_attn_qkvpacked_func
    return FlashAttnQKVPackedFunc.apply(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
 (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:110.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/home/otto/Development/temp/test.py", line 43, in <module>
    loss.backward()
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'FlashAttnQKVPackedFuncBackward' returned nan values in its 0th output.

Latest published version of flash-attention.

Does the same thing happen if you use standard implementation of attention?

use_flash_attn=False

Then it works fine @tridao

@otto-dev @tridao any updates? How to fix it?

@tridao I can reproduce this error after a very short time using the script below. Is this a user-side usage error?
import torch
import torch.optim as optim
import torch.nn.functional as F
from functools import partial
from flash_attn.modules.block import Block
from flash_attn.modules.mha import MHA
from flash_attn.modules.mlp import Mlp

embed_dim = 256 # !!! change this to 64 and the error will not be observable !!!
batch_size = 16
num_heads = 8
seq_length = 512
dim_feedforward = 1024
learning_rate = 0.01
device = torch.device("cuda")
torch.set_default_dtype(torch.bfloat16)
torch.autograd.set_detect_anomaly(True)

# Initialize model
model = Block( # TransformerEncoderLayer
        embed_dim,
        mixer_cls=partial(
            MHA,
            num_heads=num_heads,
            use_flash_attn=True,
            rotary_emb_dim=0,
        ),
        mlp_cls=partial(Mlp, hidden_features=dim_feedforward),
        resid_dropout1=0.0,
        resid_dropout2=0.0,
        prenorm=False,
    ).to(device)

optimizer = optim.Adam(model.parameters(), lr=learning_rate)
inputs = torch.full((batch_size, seq_length, embed_dim), 1000.0, device=device)

# Training loop
for i in range(999999):
    print(f'Iteration {i + 1}')
    optimizer.zero_grad()
    output = model(inputs)
    loss = output.mean()
    loss.backward()
    optimizer.step()

Output:
/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py:959: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  def forward(
/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/ops/triton/layer_norm.py:1018: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  def backward(ctx, dout, *args):
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
Iteration 10
Iteration 11
Iteration 12
Iteration 13
Iteration 14
Iteration 15
/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/autograd/graph.py:825: UserWarning: Error detected in FlashAttnQKVPackedFuncBackward. Traceback of forward call that caused the error:
  File "/home/otto/Development/temp/test.py", line 41, in <module>
    output = model(inputs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/modules/block.py", line 195, in forward
    mixer_out = self.mixer(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/modules/mha.py", line 670, in forward
    context = self.inner_attn(qkv, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/modules/mha.py", line 122, in forward
    return flash_attn_qkvpacked_func(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 729, in flash_attn_qkvpacked_func
    return FlashAttnQKVPackedFunc.apply(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
 (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:110.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/home/otto/Development/temp/test.py", line 43, in <module>
    loss.backward()
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
    torch.autograd.backward(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "/home/otto/Development/temp/venv/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'FlashAttnQKVPackedFuncBackward' returned nan values in its 0th output.

Latest published version of flash-attention.

Is there any progress?
I meet the similar issue in my self-code, so I search and run into this issue.
I tries to debug from this code snippets and my own training code. Here is the environ and some testing

hardware-env: A100 with debian 5 in linux 5.4, Driver Version: 470.129.06   CUDA Version: 12.4
torch version and flash-attn version: pytorch2.5.1 + flash-attn 2.7.2  or pytorch 2.3.2 + flash-attn 2.5.8
the trying I have done:
use flash-attn, deterministic=False, the code snippets by @otto-dev will raise nan error in FlashAttnQKVPackedFunc with  about 100 rounds(the iteration number is not stable in different run), and in my own model training code, it will also raise nan error in  FlashAttnQKVPackedFunc but in the first round.  Besides, If I run the same training code for my model repeatly 2~5 times, the nan error will disappear, and the training loop will work very nice in about 1w training iters up to now.
use flash-attn, deterministic=Ture the code snippets by @otto-dev will raise nan error in NativeLayerNormBackward0 rather than flash-attn with about 100 rounds(the iteration number is not stable in different run). In my own model training code, it will work very vice in about 5k  training iters up to now

from those phenomenon mentioned above, I guess the  undeterministic cuda kernel in flash-backward resulted in the nan in backward. I hope it would help @tridao . And I will continue working on this issue, so could I open a PR if I find the bug?


Allow users to provide custom serializer foractor states,

This is tracked by other issues related to serialization

This is tracked by other issues related to serialization


Don't enable tracing features by default

Currently the "log" feature of the "tracing" dependency is enabled (ref):
[dependencies.tracing]
version = "0.1.17"
features = ["log"]

If I understand correctly, this means that any application using "isahc" will have the tracing "log" feature enabled, with no way to disable it (because cargo features are additive). This will then not only apply to "isahc" but to all other usages of "tracing".
If that is correct, shouldn't this be disabled by default, and only enabled if the application decides to do so?

Currently, yes this is the case.

If that is correct, shouldn't this be disabled by default, and only enabled if the application decides to do so?

Is there a particular problem you are having with this? All this feature does is cause events to be delegated to the log crate if no tracing subscriber is registered, which itself will also do nothing unless a logger is registered. I see no downside to enabling this like we do now, but the downside for conditionally enabling this is no logs for people using log and not tracing who disable default features. For example, if I depend on Isahc like this:
[dependencies.isahc]
version = "1.0"
default-features = false
features = ["json"]

As a consumer I might be surprised to discover that no logs are emitted when using a log logger, but logs are emitted when using a tracing subscriber.

Currently, yes this is the case.

If that is correct, shouldn't this be disabled by default, and only enabled if the application decides to do so?

Is there a particular problem you are having with this? All this feature does is cause events to be delegated to the log crate if no tracing subscriber is registered, which itself will also do nothing unless a logger is registered. I see no downside to enabling this like we do now, but the downside for conditionally enabling this is no logs for people using log and not tracing who disable default features. For example, if I depend on Isahc like this:
[dependencies.isahc]
version = "1.0"
default-features = false
features = ["json"]

As a consumer I might be surprised to discover that no logs are emitted when using a log logger, but logs are emitted when using a tracing subscriber.

I'm using isahc through opentelemetry-jaeger's "isahc_collector_client" feature flag, so I can't set default-features = false (see https://github.com/rust-lang/cargo/issues/2823). I guess the same could be the case for other libraries.

As a consumer I might be surprised to discover that no logs are emitted when using a log logger, but logs are emitted when using a tracing subscriber.

I agree completely for isahc. But unfortunately the decision whether traces should be sent to logging seems to be a global one. So if "isahc" decides to do that, then it decides it for any other libraries in the current project, and the project itself too. So once I have "isahc" in my dependencies I cannot decide anymore if I want the "log" feature in my application or not.

I'm using isahc through opentelemetry-jaeger's "isahc_collector_client" feature flag, so I can't set default-features = false (see https://github.com/rust-lang/cargo/issues/2823). I guess the same could be the case for other libraries.

As a consumer I might be surprised to discover that no logs are emitted when using a log logger, but logs are emitted when using a tracing subscriber.

I agree completely for isahc. But unfortunately the decision whether traces should be sent to logging seems to be a global one. So if "isahc" decides to do that, then it decides it for any other libraries in the current project, and the project itself too. So once I have "isahc" in my dependencies I cannot decide anymore if I want the "log" feature in my application or not.

Sorry if this is too direct, but:

I cannot decide anymore if I want the "log" feature in my application or not.

Why do you want to decide this?

Sorry if this is too direct, but:

I cannot decide anymore if I want the "log" feature in my application or not.

Why do you want to decide this?

I agree it seems like tracing's "log" could be a sane default :D But I have the feeling if "tracing" decides to make this feature optional, then "isahc" should not enable it if it doesn't explicitly require it.
In this case my application is using tracing and logging for different use cases, and I don't want to mix the two, so I never want traces sent to the logger.
Feel free to close this ticket if you disagree!
I can set the "max_level_off" feature flag to disable tracing completely, instead of not adding a subscriber, then I won't have logs either.

I agree it seems like tracing's "log" could be a sane default :D But I have the feeling if "tracing" decides to make this feature optional, then "isahc" should not enable it if it doesn't explicitly require it.
In this case my application is using tracing and logging for different use cases, and I don't want to mix the two, so I never want traces sent to the logger.
Feel free to close this ticket if you disagree!
I can set the "max_level_off" feature flag to disable tracing completely, instead of not adding a subscriber, then I won't have logs either.

In this case my application is using tracing and logging for different use cases, and I don't want to mix the two, so I never want traces sent to the logger.

Interesting, I don't think I've heard that use-case before. In this scenario, would you want Isahc to send records to tracing, or to logs? Or neither?

Feel free to close this ticket if you disagree!

I feel like I don't have enough information yet to agree or disagree, I'd probably have to think about it more deeply. That said, changing this would be a backwards-incompatible change regardless and would likely need to be postponed to a version 2.0, whenever that happens.

In this case my application is using tracing and logging for different use cases, and I don't want to mix the two, so I never want traces sent to the logger.

Interesting, I don't think I've heard that use-case before. In this scenario, would you want Isahc to send records to tracing, or to logs? Or neither?

Feel free to close this ticket if you disagree!

I feel like I don't have enough information yet to agree or disagree, I'd probably have to think about it more deeply. That said, changing this would be a backwards-incompatible change regardless and would likely need to be postponed to a version 2.0, whenever that happens.

In this scenario, would you want Isahc to send records to tracing, or to logs?

Probably to logs.
I guess the more common use case is to use "log" and "tracing" for the same purpose, and then indeed the current behaviour makes sense.
Maybe a good solution would be a macro that can be used to produce logs and tracing events at the same time. Then each producer of tracing events can decide if they want to also produce logs or not, and it's not a global flag. But that's probably something that should be discussed on the "tracing" repo.

In this scenario, would you want Isahc to send records to tracing, or to logs?

Probably to logs.
I guess the more common use case is to use "log" and "tracing" for the same purpose, and then indeed the current behaviour makes sense.
Maybe a good solution would be a macro that can be used to produce logs and tracing events at the same time. Then each producer of tracing events can decide if they want to also produce logs or not, and it's not a global flag. But that's probably something that should be discussed on the "tracing" repo.

I've added this to the 2.0 milestone, I think I agree that this is an issue and should be re-thought. It seems like the intent of the log feature in tracing was for libraries to enable it by default, but in practice it does cause issues.
So here's the plan: the tracing dependency will become an optional dependency entirely, and not enabled by default. When enabled, Isahc will emit log entries as tracing events instead. The log feature will never be enabled and it will be up to users to enable it in their application if they want.
Unfortunately I'm not sure of any other way of allowing an application to control whether Isahc emits logs via log or via tracing when Isahc is a transitive dependency; if there were then I'd be open to it.
This has to wait until 2.0 since changing the default feature set is a breaking change.


Add github integrations

Waiting for having access to following github integrations:

[ ] Codecov ( @krzysztofzuraw needs organization access)
[ ] Read the docs ( @krzysztofzuraw needs organization access)
[ ] Pyup.io ( @krzysztofzuraw needs organization access + admin of KlubJagiellonski org needs to approve this app)

@gewaraz any progress?


GoFiber Middleware

Hello @eko I was wondering if you have thought about making a contrib GoFiber middleware for authz?

Hi @gaby,
For sure it will be interesting to have one :)
If you want to work on it do not hesitate, elsewhere I will look to implement it as soon as possible.
Maybe we could push it on https://github.com/gofiber/contrib?

@eko Agree, contrib repo would be the perfect place. If you can come up with a proof of concept we can help develop it further into a production middleware.
Thanks!


Use of Unix Domain Sockets instead of TCP Loopback Sockets

https://blogs.msdn.microsoft.com/commandline/2018/02/07/windowswsl-interop-with-af_unix/
https://github.com/MicrosoftDocs/WSL/tree/live/wsl-samples/afunix-wsl-interop

@tcyrus Thank you.
For sure LoWe will move to Unix Domain Sockets soon. LoWe officially supports the most recent stable release of W10 and WSL and not insider builds.

Thanks for the reply. This feature was added in build 17093 which is not a stable release yet.

Now that we're a major release beyond this point, any plans to move forward with that?

@ericblade
Oh yes.
At the time when 17093 was released I already checked the possibilities. As you know the Windows-side is a .NET application, and Unix Domain Sockets are still not supported out of the box in .NET. I tried to find some cool NuGet packages, etc. but no success.
If you could recommend me a nice UDS implementation for .NET I'd be more than happy.

https://stackoverflow.com/questions/40195290/how-to-connect-to-a-unix-domain-socket-in-net-core-in-c-sharp


fix(remix-dev): do not interpret JSX in .ts files

Fixes #7295

Can confirm 2.0.0-pre.4 resolved this

Can confirm 2.0.0-pre.9 working in production ✅ https://github.com/manawiki/core/commit/d15667066cf2ed0c597bf078968e02d0b70fb638
Is v2 is almost out of the oven?


Fix typo in docs/debugging.rst

docs/debugging.rst:72: controled ==> controlled
Checklist:

[X] Add tests that demonstrate the correct behavior of the change. Tests should fail without the change.
[X] Add or update relevant docs, in the docs folder and in code.
[ ] Add an entry in CHANGES.rst summarizing the change and linking to the issue.
[X] Add .. versionchanged:: entries in any relevant code docs.
[ ] Run pre-commit hooks and fix any issues.
[ ] Run pytest and tox, no tests failed.

Please rebase to 2.0.x branch, docs changes go there


Support newline-delimited JSON for seeds

Describe the feature
Allow dbt users to load in data from newline-delimited JSON files, rather than just CSV.
CSV is barely a format, and has lots of ambiguities around quoting, names, and so forth. Supporting newline-delimited JSON would make things
Describe alternatives you've considered

handling this outside of dbt (what we do now)
converting JSON data to CSV as part of the build process (a bit of a faff)
supporting other formats, like Avro (not likely to store binary data in a git repo)

Additional context
Loading in newline-delimited data into BigQuery is really easy: there's an API for it.
Who will this benefit?
Anyone looking into seeds who already has their data in newline-delimited JSON format (hi!). Anyone who has ever tried to debug dodgy quoting issues in CSV.

hey @jml - check out this issue for some prior art: https://github.com/fishtown-analytics/dbt/issues/2276
I'm going to close this issue out as a dupe, but this is a good suggestion and I'm keen to support non-csv seeds in the future!

hey @jml - check out this issue for some prior art: #2276
I'm going to close this issue out as a dupe, but this is a good suggestion and I'm keen to support non-csv seeds in the future!

@drewbanin Are you saying this request is a dupe of #2276 ? I don’t understand the relationship. Asking for newline-delimited JSON as a seed format doesn’t seem related to Jinga blocks at all.

Hey @stewartbryson - the issue I linked to includes an example of a newline-delimited json seed file. The included example looks like:
{% seed my_json_seed type=json %}

{"col1": "value", "col2": "other value"},
{"col1": "value2", "col2": "other value2"},

{% endseed %}

I do think there's an opportunity to support newline-delimited json seed files without building jinja seed blocks, but it's probably not a change we're going to prioritize in isolation. Can you tell me more about what you're looking to use newline-delimited json seed files for? That might help us better prioritize an issue like this one

@jml Would you be able to share how you currently handle it outside of dbt? I agree this would be useful, it could help around csv quoting issues, readability (if there was support for other delimiters), and representing unstructured data

Sure. We use BigQuery-specific APIs.

def load_file_to_bigquery(
    client: bigquery.Client,
    source_file: IO[bytes],
    destination_table: str,
    load_config: bigquery.LoadJobConfig,
    rewind: bool = False,
) -> LoadJob:
    logging.info("Starting load into %s in BigQuery", destination_table)
    job = client.load_table_from_file(
        source_file, destination_table, job_config=load_config, rewind=rewind
    )
    logging.info("Loading data into BigQuery")
    duration = _wait_for_job(job)
    logging.info(
        "Done in %0.1f seconds! %s rows and %s bytes were loaded",
        duration,
        job.output_rows,
        job.output_bytes,
    )
    return job

def _wait_for_job(job: bigquery.LoadJob) -> float:
    start_time = time.monotonic()
    try:
        # Starts the load asynchronously and polls until completed, raising an exception in case of problems
        job.result()
    except BadRequest:
        logging.error(
            "Errors occurred during loading: %s", job.errors, extra={"errors": job.errors}
        )
    return time.monotonic() - start_time

def load_json_line_data_to_bigquery(
    project: str,
    location: str,
    destination_table: str,
    filename: str,
    table_description: str,
    field_descriptions: Dict[str, str],
) -> None:  # pragma: no cover
    """Load line-separated JSON files from data/ to BigQuery."""
    client = get_bigquery_client(project=project, location=location)
    path = config.RAW_DATA_PATH.joinpath(filename)
    with path.open(mode="rb") as f:
        load_file_to_bigquery(
            client=client,
            source_file=f,
            destination_table=destination_table,
            load_config=bigquery.LoadJobConfig(
                autodetect=True,
                source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON,
                write_disposition=bigquery.WriteDisposition.WRITE_TRUNCATE,
            ),
        )
    update_schema_documentation(client, destination_table, table_description, field_descriptions)

def update_schema_documentation(
    client: bigquery.Client,
    table_name: str,
    table_description: str,
    field_descriptions: Dict[str, str],
) -> None:  # pragma: no cover
    """Update the documentation for a BigQuery table.

    Known limitations:

    - Does not support nested field definitions

    Feel free to fix these limitations!
    """
    table = client.get_table(table_name)
    table.description = table_description
    table.schema = _document_schema(table_name, table.schema, field_descriptions)
    client.update_table(table, ["description", "schema"])

def _document_schema(
    table_name: str, schema: Sequence[SchemaField], field_descriptions: Dict[str, str]
) -> List[SchemaField]:
    """Create a documented version of the given BigQuery schema."""
    existing_fields = set()
    new_schema = []
    for field in schema:
        description = field_descriptions.get(field.name)
        new_schema.append(
            SchemaField(
                name=field.name,
                field_type=field.field_type,
                mode=field.mode,
                description=description,
                fields=field.fields,
            )
        )
        existing_fields.add(field.name)
    undescribed_fields = existing_fields - set(field_descriptions)
    # TODO: Also raise an exception if we have described fields that don't exist.
    if undescribed_fields:
        # TODO: Raise a more specific exception so we don't have to pass table_name in.
        raise RuntimeError(f"Unexpected fields defined in {table_name}: {undescribed_fields}")
    return new_schema

Happy to answer any questions. Memrise Limited is making this code available under the Apache License 2.0.

If we're willing to stick with agate as our dataframe provider—very much an open question, since there are some very good reasons to rip it out (#3413)—but if we were to stick with it, I think this might be a fairly trivial addition, since agate supports a from_json method that looks quite similar the from_csv method we use for seeds today.
We'd need to update the load_agate_table provider method, and/or turn the agate_helper.from_csv method into a generalized from_file method.
Those are implementation details. I'd still like to hear a bit more from users about the rationale for supporting ndjson—ease of quoting/escaping? ease of debugging?—rather than requiring conversion to csv as a preprocessing step.

I'd still like to hear a bit more from users about the rationale for supporting NDJSON—ease of quoting/escaping? ease of debugging?—rather than requiring conversion to CSV as a preprocessing step.

For us it's about having a harder-to-get-wrong format. It's really easy to get CSV wrong (mostly quoting/escaping, as you say).
There's also a migration factor. Before we switched to dbt, we were loading data from NDJSON files. Now, switching to dbt, we have to make those files worse (FSVO 'worse'), which feels off, given that everything else about dbt has been an improvement.

Our use case:
We JSON in our "seeds" in order to reflect some nested relationships. Something like one row per user containing a JSON list of groups or roles that this user belongs to. We do not want to flatten such lists immediately but rather keep them as VARIANT or ARRAY type.
Currently we maintain such JSON "seeds" as a string literal of a JSON array of these objects, parse them via Snowflake's parse_json() and flatten() the outer array.
It would be much more convenient to be able to use NDJSON here, so the editor can provide syntax highlighting etc.
However, as said, our use case requires nested lists and maps to stay nested.

If a flat table is the result that we are aiming for after loading the seed, we could live with the nested fields being included as escaped text, though, as that could be easily converted downstream.
So, yes, type and quoting safety are the big benefits of JSON here. CSV is error-prone and too less standardized imho.
Totally agree with @jml here.

Same as what some others have mentioned. We use bigquery, and miss not having support for nested data in seeds.
One use case was testing where it would have been nice if the seeds mirrored the structure of what was being tested - without using nesting in seeds it added an extra step to first create that nested structure from csv read.

Hello,
We have JSONB columns in our Postgres database that we use as a source for our data. Managing JSON inside CSVs is painful where as having it inside NDJSON or YAML will make it much easier to manage. That is our use case for wanting something other than CSV as the seed file format.

We use Snowflake source tables that heavily use VARIANT columns. Using CSV seeds to add sample data for testing transformations is resulting in error SQL compilation error: Expression type does not match column data type, expecting VARIANT but got VARCHAR(210) for column PAYLOAD. I am not entirely sure how to use the seed json example given above inside sources.yml file as the tables are already created in our EL (extract and load) process and we don't define them as models in dbt.
We are overriding the snowflake load macro snowflake__load_csv_rows to use parse_json for variant columns but it seems like an ugly hack. I am not sure if support of ndjson will resolve our issue but a way to load text into Snowflake variant columns will be quite useful.

@naga-kondapaturi-zefr Are you trying to use seed-related functionality for a table that already exists in your data warehouse? Or in order to load a CSV containing a column with a JSON value? (Perhaps loading it as a string and casting it within a staging model?)

I worked with a dbt user recently who uses Snowflake, who had an interesting way to solve the lack of JSON seeds (ND or otherwise):
They create dbt models that use the PARSE_JSON function, to persist VARIANT type tables, that they can use as static datasets. For example:
# models/animals.sql
select parse_json('{"cats": "the best", "dogs": "the worst"}') as json_blob

@drewbanin is this one on the roadmap? This would be a helpful feature for Meru for the same reason @jml mentioned:

For us it's about having a harder-to-get-wrong format. It's really easy to get CSV wrong (mostly quoting/escaping, as you say).

We use Snowflake source tables that heavily use VARIANT columns. Using CSV seeds to add sample data for testing transformations is resulting in error SQL compilation error: Expression type does not match column data type, expecting VARIANT but got VARCHAR(210) for column PAYLOAD. I am not entirely sure how to use the seed json example given above inside sources.yml file as the tables are already created in our EL (extract and load) process and we don't define them as models in dbt.
We are overriding the snowflake load macro snowflake__load_csv_rows to use parse_json for variant columns but it seems like an ugly hack. I am not sure if support of ndjson will resolve our issue but a way to load text into Snowflake variant columns will be quite useful.

@naga-kondapaturi-zefr would you be able to share the override implementation of snowflake__load_csv_rows to use parse_json for variant columns. We are facing the exact same problem.

Anyone in need of the hack solution in overriding snowflake__load_csv_rows here it is:

{% macro snowflake__load_csv_rows(model, agate_table) %}
    {% set batch_size = get_batch_size() %}
    {% set cols_sql = get_seed_column_quoted_csv(model, agate_table.column_names) %}
    {% set bindings = [] %}
    {% set column_list = [] %}
    {% set variant_columns = [] %}
    {% set statements = [] %}

    {#  Get all columns that have type variant  #}
    {% for i in model['config']['column_types'].items() %}
        {% if i[1] == 'variant' %}
            {% do variant_columns.append(i[0]) %}
        {% endif %}
    {% endfor %}

    {#  Create column list expression for columns using parse_json for variant types  #}
    {% for i in agate_table.column_names %}
        {% if i in variant_columns %}
            {% do column_list.append('parse_json(column{0})'.format(loop.index)) %}
        {% else %}
            {% do column_list.append('column{0}'.format(loop.index)) %}
        {% endif %}
    {% endfor %}

    {#  Create insert query  #}
    {% for chunk in agate_table.rows | batch(batch_size) %}
        {% set bindings = [] %}

        {% for row in chunk %}
            {% do bindings.extend(row) %}
        {% endfor %}

        {% set sql %}
            insert into {{ this.render() }} select {{ ', '.join(column_list) }} from values
            {% for row in chunk -%}
                ({%- for column in agate_table.column_names -%}
                    %s
                    {%- if not loop.last%},{%- endif %}
                {%- endfor -%})
                {%- if not loop.last%},{%- endif %}
            {%- endfor %}
        {% endset %}

        {% do adapter.add_query('BEGIN', auto_begin=False) %}
        {% do adapter.add_query(sql, bindings=bindings, abridge_sql_log=True) %}
        {% do adapter.add_query('COMMIT', auto_begin=False) %}

        {% if loop.index0 == 0 %}
            {% do statements.append(sql) %}
        {% endif %}
    {% endfor %}

    {# Return SQL so we can render it out into the compiled files #}
    {{ return(statements[0]) }}
{% endmacro %}

{% materialization seed, adapter='snowflake' %}
    {% set original_query_tag = set_query_tag() %}

    {% set relations = materialization_seed_default() %}

    {% do unset_query_tag(original_query_tag) %}

    {{ return(relations) }}
{% endmaterialization %}

I'd still like to hear a bit more from users about the rationale for supporting NDJSON—ease of quoting/escaping? ease of debugging?—rather than requiring conversion to CSV as a preprocessing step. Unless I'm missing something big, I don't think JSON seeds would have a leg up in terms of support for semi-/unstructured data, since the ultimate aim is always conversion to tabular format (data frame).

@jtcohen6 , my use case is very similar to what @naga-kondapaturi-zefr mentioned above.
I have a raw table in Snowflake that receives events through Kafka from an operational system. The table has only two columns, both of VARIANT type.
create table if not exists PROD.RAW.EVENTS (
	RECORD_METADATA VARIANT,
	RECORD_CONTENT VARIANT
);

The first column is generated by a Snowflake Connector for Kafka and the second contains the actual JSON events. We are using dbt to extract the info that we need from events and transform it to analytics. Now, I need to create an environment for testing. It is located in a separate Snowflake account, so I can't clone the original raw table. I was looking for options on how to load JSON to Snowflake, and when I learned about dbt seed I thought that it would be the perfect tool for the job, but then it was disappointing to realize that only CSV was supported (and I'm getting the same SQL compilation error).
One thing to mention here. In my use case I have a huge JSON event, with lots of nested levels of varying depth, that should be loaded to the VARIANT column as they are, without being parsed and loaded to specific columns, as is the case for CSV files (and I assume is the case for some people in this thread that would like to get this feature). Perhaps that's is something to bear in mind for any person who would be working on this feature, and load the data depending on the destination table (VARIANT datatype or not).
This message seems to have irrelevant details, but it's for the purpose of providing more context and understanding the use case.

@jtcohen6 what is the current status of considering supporting JSON seeds? This issue is pretty long so I've not read all of the proposals / use cases but it seems to me like a very reasonable and generic thing to do to want to load seed data that contains nesting. At which point, you're dead out of the water with our primitive friend, CSV.

Not sure if its worth adding another issue for this but YAML as well would be good here for multiline strings.
eg I want to store prompts in a database and use seeds to add them, yaml would look something like this
- id: 9ryuf89wh
  prompt: |
       your task is to do x
       you should do it via y
       here is the data: {data}
- id: i8d3h89
  prompt: |
       your task is to do c
       you should do it via b
       here is the data: {data}

I removed the stale label to keep this issue open.

Commenting on the issue to keep it open. We'd still like this functionality.

Bumping up against this now trying to add seed data for our CI environment


Use Buffer.from() instead of Buffer constructor

Fixes #10

Thanks for this, appreciated!

No problem. Thanks for your excellent work.


Consume the simplified controller constructor.

See also: knative/pkg#427
/hold
This is RFAL, but shouldn't merge until the upstream change lands and this is reworked to consume it.

@ImJasonH I think the e2e test failure may be because Build is way behind on knative/pkg.  Can you TAL?

@ImJasonH I think the e2e test failure may be because Build is way behind on knative/pkg. Can you TAL?

Just getting back to this now. It looks like the webhook controller isn't being registered, so no validation is happening at all:
{"level":"error","logger":"webhook","caller":"webhook/webhook.go:296","msg":"failed to register webhook","knative.dev/controller":"webhook","error":"failed to fetch our deployment: deployments.apps \"build-webhook\" is forbidden: User \"system:serviceaccount:knative-build:build-controller\" cannot get deployments.apps in the namespace \"knative-build\"","stacktrace":"github.com/knative/build/vendor/github.com/knative/pkg/webhook.(*AdmissionController).Run\n\t/usr/local/google/home/jasonhall/go/src/github.com/knative/build/vendor/github.com/knative/pkg/webhook/webhook.go:296\nmain.main\n\t/usr/local/google/home/jasonhall/go/src/github.com/knative/build/cmd/webhook/main.go:92\nruntime.main\n\t/usr/lib/google-golang/src/runtime/proc.go:200"}

Do you know any recent changes to knative/pkg that might be the culprit? Tests pass at knative/build head, using knative/pkg from March.

@ImJasonH I bet is was a change from beta apps to v1 apps, so changing the RBAC should fix this.

Superceded by #615


Add data which allows users to select types of commodities

For the Mines and Mineral deposits layers, users want to be able to select specific minerals to be visualised (eg. lithium, iron ore, gold etc.).

[ ] customer to send a list of 10-20 priority minerals

[ ] create a new layer with minerals as displayed variables

Any word on the list of 10-20 priority minerals, @AnaBelgun ?

This is for next release; they are still deciding which minerals.

So I got it showing all the points from the WMS service: https://terria-cube.terria.io/#share=s-4qz4BfvCliAx9wTR1NuvFTSJgQE
GetFeatureInfo doesn't work though, I've not investigated that yet.
Technically I think we could already support the 2.a. option of "supporting" SLD's by creating separate layers and using the parameters block to specify the SLD. That's a big bunch of layers with a lot of SLD that would end up encoded in our init file though...
Option 1 doesn't seem very attractive for the reasons you gave.
Option 3 isn't great because we don't want to hold on to the data.
Note that there is also a WFS service at that URL, but it's slow and I couldn't get Terria to display anything even though it looked like it did return the data.
Out of these options I actually like 2.b. the most.

Yeah that was my thought too @meh9: construct the SLDs at catalog build time and stuff them in the init file as parameters. It might even be possible - perhaps with a small code change - to trick terria into thinking the different SLDs are WMS dimensions, and then it would create UI on the workbench to let the user pick from them. There would be a whole lot of SLDs in the init file though. GZIP should mostly make it ok.
Not sure what you mean by 2.b though? Is that GA pre-generating the SLDs and letting us reference them by name?

@meh9 and @kring would you mind finalising the investigation on this? let me know if you need to contact this team at GA

@AnaBelgun yes, let's ask GA if they can create the SLD's as different WMS Styles on the server, which would make all WMS clients jobs easier.

@stevage @meh9 @kring
Michael Sexton here, technical custodian for the AusGIN Portal. I can address a few of your questions about how the portal works and what is the best way to interact with out underlying services
Firstly, the non-WMS API call made is a workaround developed to allows us to send POST WMS GetMap and GetFeatureInfo requests. The ability to POST to WMS was supported in OpenLayers 2, but removed by OpenLayers 4. In retrospect our solution is not the ideal way of doing things, it would be better to have modified our client-side interactions to allow POST requests for WMS to underlying services, but server side API calls were developed first by our partners at AuScope and we ran with that.
We use SLDs to filter the data, and there are three ways by which to do this:

Send the online SLD resource to the service as a URL
Send the SLD_BODY in a GET request
Send the SLD_BODY in a POST request

The first is usually not possible due to corporate sysadmins locking down port 80 on their infrastructure, the second runs into problems if the SLD_BODY gets too large. And for our purposes, the SLD_BODY can often get quite large. An example:
http://api.portal.geoscience.gov.au/mineralOccurrenceViewFilterStyle.do?commodityUri=http%3A%2F%2Fresource.geosciml.org%2Fclassifier%2Fcgi%2Fcommodity-code%2Fproduced-commodity&timescaleUri=http%3A%2F%2Fresource.geosciml.org%2Fclassifier%2Fics%2Fischart%2FPhanerozoic&serviceUrl=https%3A%2F%2Fgs.geoscience.nsw.gov.au%2Fgeoserver%2Fwfs
So the option we use to interact with our service is to POST the SLD_BODY to it. Even with a large request like the above, it is quite performant.
GetFeatureInfo for GeoServer also accepts SLD_BODY as a POST request, which means you can send the SLD generated above and interact with a point-click.
The filtering by commodity is done by sending URIs to the service from a controlled vocabulary (http://vocabs.ga.gov.au/cgi/sissvoc/commodity-code/concept.html). For your purposes you could pre-construct the SLDs you need for the commodities you want to select and POST these to our service.
Let me know if you have any more questions.

@michaelsexton
Could you explain a little more about Option 1 please?
It would be relatively trivial for us to host SLD's in an AWS S3 web bucket and send the URL to each SLD as a URL query parameter to the WMS with each access, is that how that would work, or did I misunderstand?
If that is it, have you considered (or do you already have) an online repository of pre-canned SLD's that we could just point to so we don't have to maintain our own?

@meh9  That's how it would work, yes.
Currently our web service blocks port 80 because it is on our corporate servers. We are intending on migrating it to the cloud in the next month. This will allow sending SLDs as URLs.
It is possible for us to provide a pre-canned SLDs for you. However, if we were to go down the self-hosting route, it would be more ideal for us to set these as pre-defined styles on our service, which you simply request using the STYLES parameter.
What would be the set of commodities you would need for your requests?

@michaelsexton in that case I definitely think this option is the best one. :)
Of course we are much happier for you to host the SLD's - as STYLES would be preferable of course, but we can also work with URL's if you decide to host them.
Failing that we would be ok with hosting the SLD's ourselves, but then they will inevitably become outdated at some point in the future...
@AnaBelgun can you confirm a list of commodities please? Is it the list above in https://github.com/TerriaJS/InvestorMap/issues/150#issuecomment-454581545 ?

@michaelsexton ideally they want all commodities available in your service; they are interested in replicating as much as possible the filters for the list of commodities, categories, reserves etc.

Bit of a long shot, but @michaelsexton is it perhaps possible to expose the three drop-downs (Commodity, Resource Category, Provider) as WMS dimensions? Not provider, I guess, because that switches between two separate WMS servers, right?

@AnaBelgun - all the commodities available in the service is fairly extensive, there are nearly 300! (http://api.portal.geoscience.gov.au/getAllCommodities.do)
I'm a bit unsure as to how you would implement it - our API interfaces with a vocabulary server which is the front end of a triple store that contains SKOS vocabularies. Our list of commodities are hierarchical - that is if you select 'precious metal', we use a reasoner to determine you'll want gold, silver, platinum group metals (which is also the top of a hierarchy of platinum, palladium, osmium, etc).
I had thought a set of pre-canned styles based on broad categories would solve your problem, eg 'has resources, critical commodities', 'has reserves, base metals', etc. Nothing too granular.
@kring not sure what you mean, but I think the quick answer is that it is possible with GeoServer using their cql_filter parameter:
https://docs.geoserver.org/stable/en/user/tutorials/cql/cql_tutorial.html

@michaelsexton you mean all "primary commodities", in the drop down list from agate to zirconium are 300 in total? those are the ones I was referring to.

@AnaBelgun yes, 292 to be exact, and growing.
Depending on what you need to deliver, I usually think it is preferable in most cases to group them - base metals, precious metals, rare earths, bulk commodities, energy minerals, mineral sands, etc, with some major/important commodities as standouts (copper, gold, iron ore, coal, etc).
We have the full list because we can take a broad term like "base metals" and use our reasoner to infer what the user meant:
http://api.portal.geoscience.gov.au/mineralOccurrenceViewFilterStyle.do?commodityUri=http%3A%2F%2Fresource.geosciml.org%2Fclassifier%2Fcgi%2Fcommodity-code%2Fbase-metal&serviceUrl=http://services.ga.gov.au/earthresource/wms

@michaelsexton ok, I'll discuss with Austrade and check this option (groups + some popular indiv. ones)

@michaelsexton hi. Austrade team will check the commodities list and draft a list of the most relevant/important based on your advice: have a list with the high-level groups and some popular individual ones, such as copper, gold, iron, coal etc.
To both teams (minerals and terria): @michaelsexton @meh9 @kring
Just to confirm: Austrade would like for the users to be able to have this list of 20 max minerals and be able to select to view, single, multiple or all of them on the map. The information available for each, would include the reserves.
They asked if possible to also show all 292 ones on a single layer as well (either part of this layer, or a separate one). They understand it won't be possible to select one, or multiple as part of the 292 layer, but part of the 10-20 Commodities layer (described above).

Hi @michaelsexton here is the list of the commodities from Austrade for the pre-canned styles:
There are many ways to cut this and if GA wants/needs to regroup or combine some that is OK. If you have other suggestions in terms of grouping them, let us/Russell know please.
•	Iron Ore
•	Coal black
•	Coal brown
•	Gold/Silver
•	Copper
•	Nickel
•	Lead
•	Zinc
•	Bauxite/alumina
•	Lithium
•	Mineral sands (rutile,  ilmenite, zircon)
•	Graphite
•	Cobalt
•	Manganese
•	Uranium
•	Phosphate
•	Vanadium
•	Platinum Group Elements
•	Rare Earth Elements

Hi @AnaBelgun
Generating the above SLDs for commodities will be easy, where it gets hosted is another matter.
Are there any colour codes or icons you want?
I can generate the SLDs and provide them to you. It will not be able to be hosted with the service until the end of next month. If these timelines don't suit let me know.

HI @michaelsexton thanks. Austrade confirmed that we'll wait for the SLDs to be hosted with the service and include that in the March InvestorMap release. just to note that we'll need few days before the actual release (27-28 March) so we can build the new version and get it checked & approved.

from GA team: There are now styles associated with each commodity listed above. this is the service endpoint for testing
http://<IP_ADDRESS>:8080/geoserver/wms
@meh9 @kring would you like to work on this or delegate?
It needs to go in the end of March release

Ok, the new SLDs make life very much easier. I've set up each style as its own catalog item for consistency with other layers (rather than using the style drop down control within a single catalog item).

That works fairly well. The only issue is that currently all the styles are circles of the same shade of magenta:

This will obviously be problematic for anyone who wants to layer several different kinds of resources together.

Also, could I have a little guidance on what the group should be called and where it should sit? Currently I made it "Mineral Resources and Reserves" (reflecting the WMS layer name that is being shown), near the top of the Mining group.

2019-03-25 Checking with client on:

their preference of having drop down menu vs separate layers (initially was drop down selector)
their colour preference for those commodities (they do need to change)

@stevage
re: 1) - they confirmed their preference is to have them on one layer with a drop down selector. Is that possible in the timeframe?
2) colouring - in progress

Oh, sure. Heh, one layer with a drop down selector is actually much simpler than what I implemented. It also kind of removes the need for different colours because they won't be able to see several mineral types at once.

awesome, waiting for the final colours now and then we're ready to go
or can you implement it with what colours Michael set for now
Thank you @stevage

when this sub group is ready, the catalogue should change a bit for Mining:

rename Mines and Mineral Deposits: Mines and Mineral Deposits - All and place it at the top of Mining catalogue
name the new sub group (with this curated list): Mines and Mineral Deposits - Selected Commodities
delete the Lithium Projects layer (included now in the new Selected group)

the client wants a drop down box but also the ability to show multiple commodities at once

Hmm, I really can't imagine how that would be possible. A drop down box (or at least, this style of drop down box) is by definition "choose one out of several options". Maybe they're thinking of a drop down box that contains checkboxes - which our style selector doesn't support. (And for that matter, a single WMS layer with multiple styles simultaneously combined is not, afaik, possible.)

ok, we figured in the end that it would be almost impossible; please leave the implementation as is - multiple layers

Ok, have made all the requested changes.


Update OSX Install instructions for MakeSymlinks

fixes #6202

Unfortunately no!
If your /usr/local/bin is managed by brew, you have user access to it, there is no need to use sudo!
If you use it, then just run brew doctor to fix the permissions issue but indicating here in the documentation that sudo is required is not good imo


Fix #538

This PR is a fix for issue #538
Suppose there are 2 goroutines A and B. Assume A is updating the routing table. While this happening, B comes and sees that someone is already updating the routing table, and appends its channel to get notified once A updates the table.
Happy case
If B's context is not Done(),  then A will send an empty struct{} to the channel and then B continues with its flow.
Failing case
If B's context is done before A updates the routing table, then B stops listening to the channel it just pushed to be notified on. Now once A updates the table, it goes to send empty struct{} to the channel, but this results in a panic situation because there is no one to listen on that channel anymore.
Fix
Instead of notifying the goroutines by sending an empty struct{}, we should rather just close the channel. Closing the channel is panic safe, and will continue on with the existing flow.

@StephenCathcart I have created a new PR with the user with which I signed the CLA. Please check now?

@StephenCathcart Why is it failing now? 🤔

@StephenCathcart Why is it failing now? 🤔

Hi, we have an extra security feature, a membership check, as part of our build which was failing but has now been resolved (the build is currently running). Your whitelist check is ok. I'll let you know once this has been merged and will be included in the upcoming release at the end of the month.


stage1: move more logic out of AppUnit

Try and simplify app unit generation in a few ways This is in preparation for pluggable stage2 isolation.

generate an intermediate struct with runtime information, rather than generating unit directives directly
Split systemd-specific and more generic unit code

Failures are known flake (TestAppUserGroup)

@s-urbaniak updated, thanks!

This got a conflict in the meanwhile. It looks fine in general, I'll do a last pass after the rebase.

@squeed bump

Rebased, thanks!

This PR got hit by a github downtime, so it is in a kind for weird state. @squeed Mind rebasing and pushing again?

Rebased - let's go for green!

fail :-(

Neat, it printed the byte sequence. The error message is

add: pod "b5b86b03-4e7f-4251-bc60-aed97540ef63" isn't currently running

I'll dig in to this.

Bad merge - OnFailure snuck back in. Fixed.


Added atrix to required fields

Resolves #309

Coverage decreased (-0.01%) to 47.51% when pulling a924095093f72b7671264c5914537041c9ed3e8e on dame-sprint-2021-feb-sage309-requiredFields into 782322b646dd900a2e50e99df1777d7ed29a6d13 on dame-sprint-2021-feb-staging.