Perpetual Bounty: llama Python runtime 20%+ faster on M1 -- $200+

Last bounty was $1000 for 100%, so it scales linearly. Minimum I'll consider is 20% and $200, but for 40% we'll pay $400 etc... @roelofvandijk
Gains must not come from changing LLaMA or using the JIT. Only the "ran model" time matters, the GPU sync time is a different problem that you don't have to worry about for this bounty. Will be evaluated on M1 Max python3.11 with asserts turned off as below.
Will not merge anything that negatively impacts readability or adds tons of lines. Benchmark timing is 99.58 ms, so 79.66 ms is needed to be eligible.
kafka@tubby:~/fun/tinygrad$ python3.11 -O examples/llama.py --prompt "Hello." --count 10 --temperature 0 --timing
using METAL backend
ram used: 13.48 GB, freqs_cis                                         : 100%|███████████████████████████████████████████████████████████████████████████████| 292/292 [00:04<00:00, 67.66it/s]
loaded weights in 4322.95 ms, 13.48 GB loaded at 3.12 GB/s
Hello.
ran model in 1188.19 ms
sync in 130.40 ms
 I
ran model in 185.39 ms
sync in 4.61 ms
'
ran model in 101.04 ms
sync in 3.80 ms
m
ran model in 102.47 ms
sync in 3.78 ms
 a
ran model in 109.10 ms
sync in 3.87 ms

ran model in 99.58 ms
sync in 3.88 ms
2
ran model in 104.83 ms
sync in 3.74 ms
0
ran model in 102.63 ms
sync in 3.62 ms
 year
ran model in 113.94 ms
sync in 3.75 ms
 old
ran model in 102.66 ms
sync in 3.71 ms

Minor semantic nit (if I am not mistaken):
If 200ms->100ms is 100% speedup (2x as fast), 99.58ms->79.66ms is already a 25% speedup.
A 20% speedup would be 82.98ms (99.58ms/120).
I'd like to give his a shot and see how far I can get with subtle changes and conceputal refactors.

Good point, updated

I must be doing something wrong because I'm on a 2021 MacBook Pro M1 Max and it is using the CPU: (base) ➜  tinygrad git:(master) ✗ conda activate my_conda_env_with_py311
(my_conda_env_with_py311) ➜  tinygrad git:(master) ✗ python3.11 -O examples/llama.py --prompt "Hello." --count 10 --temperature 0 --timing
using CPU backend
using LLaMA-7B model
ram used: 13.48 GB, freqs_cis                                         : 100%|█████████| 292/292 [00:00<00:00, 8887.53it/s]
loaded weights in 239.09 ms, 13.48 GB loaded at 56.37 GB/s
Hello.
ran model in 51756.77 ms
sync in 12117.49 ms
I
ran model in 46876.28 ms
sync in 11304.77 ms
'
ran model in 45436.27 ms
sync in 11041.51 ms

I saw something on simonwenkel.com that said do "conda install -c conda-forge pocl pyopencl clinfo" to use GPU. putting PYOPENCL_COMPILER_OUTPUT=1 in cache.py or llama.py doesn't seem to do anything, neither does setting that environment variable. and my ran model times are still way more than yours. 786.50 ms at best. (my_conda_env_with_py311) ➜  tinygrad git:(master) ✗ python3.11 -O examples/llama.py --prompt "Hello." --count 10 --temperature 0 --timing
using GPU backend
using LLaMA-7B model
ram used: 13.48 GB, freqs_cis                                         : 100%|██████████████████████| 292/292 [00:00<00:00, 6212.08it/s]
loaded weights in 69.41 ms, 13.48 GB loaded at 194.19 GB/s
Hello.
/Users/steve/miniforge3/envs/my_conda_env_with_py311/lib/python3.11/site-packages/pyopencl/cache.py:495: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
_create_built_program_from_source_cached(
/Users/steve/miniforge3/envs/my_conda_env_with_py311/lib/python3.11/site-packages/pyopencl/cache.py:499: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
prg.build(options_bytes, devices)
ran model in 32767.68 ms
sync in 5760.29 ms
I
ran model in 422.49 ms
sync in 962.84 ms
'
ran model in 314.56 ms
sync in 786.50 ms
m
ran model in 203.72 ms
sync in 884.54 ms
a
ran model in 292.95 ms
sync in 978.82 ms
ran model in 244.11 ms
sync in 875.56 ms
2
ran model in 212.70 ms
sync in 896.38 ms
0
ran model in 278.44 ms
sync in 833.53 ms
year
ran model in 236.25 ms
sync in 956.24 ms
old
ran model in 299.93 ms
sync in 875.82 ms
male%

after putting that variable in my .zshrc file: Hello.
/Users/steve/miniforge3/envs/my_conda_env_with_py311/lib/python3.11/site-packages/pyopencl/cache.py:495: CompilerWarning: Built kernel retrieved from cache. Original from-source build had warnings:
Build on <pyopencl.Device 'cpu' on 'Portable Computing Language' at 0x7fba3582c600> succeeded, but said:
warning: /Users/steve/.cache/pocl/kcache/tempfile_o47SpM.cl:1:26: unsupported OpenCL extension 'cl_khr_fp16' - ignoring
_create_built_program_from_source_cached(
/Users/steve/miniforge3/envs/my_conda_env_with_py311/lib/python3.11/site-packages/pyopencl/cache.py:499: CompilerWarning: From-binary build succeeded, but resulted in non-empty logs:
Build on <pyopencl.Device 'cpu' on 'Portable Computing Language' at 0x7fba3582c600> succeeded, but said:
warning: /Users/steve/.cache/pocl/kcache/tempfile_o47SpM.cl:1:26: unsupported OpenCL extension 'cl_khr_fp16' - ignoring
prg.build(options_bytes, devices)
/Users/steve/miniforge3/envs/my_conda_env_with_py311/lib/python3.11/site-packages/pyopencl/cache.py:495: CompilerWarning: Built kernel retrieved from cache. Original from-source build had warnings:
Build on <pyopencl.Device 'cpu' on 'Portable Computing Language' at 0x7fba3582c600> succeeded, but said:
warning: /Users/steve/.cache/pocl/kcache/tempfile_cSJIT0.cl:1:26: unsupported OpenCL extension 'cl_khr_fp16' - ignoring
_create_built_program_from_source_cached(
/Users/steve/miniforge3/envs/my_conda_env_with_py311/lib/python3.11/site-packages/pyopencl/cache.py:499: CompilerWarning: From-binary build succeeded, but resulted in non-empty logs:
Build on <pyopencl.Device 'cpu' on 'Portable Computing Language' at 0x7fba3582c600> succeeded, but said:
warning: /Users/steve/.cache/pocl/kcache/tempfile_cSJIT0.cl:1:26: unsupported OpenCL extension 'cl_khr_fp16' - ignoring

Can you open a discussion post for this or join the discord for help? Thanks!


Motivation and Context
This is a common mistake that's happened more than once to users. It causes the profiler to produce confusing results. Better catch the error early and notify.
How Has This Been Tested?
I ran gProfiler as a Docker container without --pid=host and got the expected message.
I checked some old CentOS 6 (kernel 2.6.32) machine and verified my assumption of kthreadd holds there as well.
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] I have updated the relevant documentation.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

Still flaky @guybortnikov :/


zombie slims

This is likely a question for @cmungall
Most of the slims I removed are cleanly gone from the ontology, however goslim_goa and gosubset_prok seem to be coming back from some import :

Any idea what's happening ? Should we just ignore this or do we need to remove these slims from [wherever they are coming from]?
Thanks, Pascale

It looks like they are coming from extensions/x-disjoint.owl

If you hover over the term, it will say where it comes from. Then if you go to the top menu bar in protege, you can switch ontologies in the drop down menu. Switch to go/imports/x-disjoint.owl.
Now search on object properties and you will see the zombie ones appearing as bold under subset property. As for whether or not it is safe to edit this file.....?

For example:
https://github.com/geneontology/go-ontology/blob/master/src/ontology/imports/x-disjoint.owl
has this:
<!-- http://purl.obolibrary.org/obo/go/extensions/x-disjoint#goslim_candida -->

<owl:AnnotationProperty rdf:about="http://purl.obolibrary.org/obo/go/extensions/x-disjoint#goslim_candida">
    <rdfs:comment rdf:datatype="http://www.w3.org/2001/XMLSchema#string">Candida GO slim</rdfs:comment>
    <rdfs:subPropertyOf rdf:resource="http://www.geneontology.org/formats/oboInOwl#SubsetProperty"/>

</owl:AnnotationProperty>


Image increasing in byte size even when the dimensions are smaller.

I've got an original png image at the following dimension - 1920 × 1080 at 12kb.
The options in my grunt config are:
    options: {
				sizes: [{
					name: "xsmall",
					width: 160,
					quality: 80
				},{
					name: "small",
					width: 320,
					quality: 80
				},{
					name: "medium",
					width: 640,
					quality: 80
				},{
					name: "large",
					width: 1024,
					quality: 80
				},{
					name: "xlarge",
					width: 1600,
					quality: 80
				}],
				upscale: false,
				createNoScaledImage: true
			}

Every time run the grunt task some of the images seem to get larger in bytes even though they're smaller in size.
image-xsmall - 2kb (smaller)
image-small - 3kb (smaller)
image-medium - 8kb (smaller)
image-large - 15kb (larger)
image-xlarge - 30kb (twice as large!!)

Could you supply the original image please Tom?
Are you using Imagemagik or Graphicsmagik?
What happens if you run the image through ImageOptim afterwards?

Closing due to lack of response.


Chess.com is broken

I've played with black figures, but it works only for white ones.
upd: seems like it remembers one step and keep advising it regardless of color.

Need more clarification.

Can't reproduce. Everything works fine.


azure-identity requires java.desktop to open browser window

It would be much better if we did't need to have a dependency on java.desktop to open a browser window. This won't always necessarily work either - e.g. having this dependency might prevent identity from working on mobile platforms. We should consider having a pluggable system or other approach to not force a java.desktop dependency directly on azure-identity.

InteractiveBrowserCredential is only intended to work on desktops.
For mobile msal4android should have all the UI designed already in their aar. No need to extend this class for mobile platforms.

Right, but the flow-on effect of having InteractiveBrowserCredential in azure-identity is that it makes the module heavier for all use cases. Headless servers deployments would need to bring in java.desktop, even though the InteractiveBrowserCredential API is not used.

java.awt.Desktop seems to be built-in in JDK 8. Are you suggesting it's removed in a future JDK release? If so - what's our alternative? We can make it a provided dependency in that case.

My proposal would be to remove the module dependency on java.desktop, and to introduce the following code into azure-identity to open urls:
private static void openUrl(String url) throws IOException {
    Runtime rt = Runtime.getRuntime();

    String os = System.getProperty("os.name").toLowerCase();
    if (os.contains("win")) {
        rt.exec("rundll32 url.dll,FileProtocolHandler " + url);
    } else if (os.contains("mac")) {
        rt.exec("open " + url);
    } else if (os.contains("nix") || os.contains("nux")) {
        rt.exec("xdg-open " + url);
    }
}

Be sure to log to client logger if none of the if/else conditional branches works and we fall through to the else.


fixed bug on development in signin with github

RFC - #830

Closing this, as this issue is irrelevant now


add dockercfg secret types

Part 1 of https://github.com/GoogleCloudPlatform/kubernetes/pull/7894.
This adds the DockerConfig secret type.
@brendandburns ptal.  This is used by the subsequent pulls for #7894, so getting this first would be convenient.

Will review API changes.

Updated to use ".dockercfg" config secret.  The first commit is in https://github.com/GoogleCloudPlatform/kubernetes/pull/8072.

I made some comments on the commit. Will review your other two PRs later, this should keep you busy for a bit.

rebased and added patch tags.

I think all the comments have been addressed.

Squashed down the commits so they're easier to read.

switched validation to use a map[string]interface{} to ensure well-formed json.

LGTM

Travis is a flake.


I can't get RawUrl.  Request.GetDisplayUrl & Request.Path is bad, any method(I can't find)

My request original url:
http://localhost/2http%3A%2F%2Flocalhost%3A803%2Fhomepage%2F3%2FSignup%3FM49HD59E
RawUrl & Request.GetDisplayUrl & context.Features.Get().RawTarget %2F%2F miss one '/', not is http://(is http:/)
after one day, i can't find  any method.
follow is good
1:  /2http://localhost:803/homepage/.....
2:  /2http%3A%2F%2Flocalhost%3A803%2Fhomepage%2F.....
asp.net core 2.0

What are you trying to do?

I want to create a Web site that can accept any text and generate a QR. I hope it can accept a URL, but I use ASP. NET Core 2.0 is not possible, because when there are 2 escaping '/', you will miss 1 '/'. I want to get the original URI of the escape.

context.Features.Get<IHttpRequestFeature>().RawTarget?

context.Features.Get().RawTarget is bad too.

(I don't want to use querystring.)

You're pretty much guaranteed to have issues trying to encode a URL as a path segment.
If you're running behind IIS in this case, I'll bet that's what's munging the path. Different servers have different quirks for this sort of thing.
@Tratcher is the expert on this.
What's your reason for not wanting to use the query string?

Yes, sounds like a known iis/http.sys behaviour. Have you tried running kestrel without IIS?

thank you, Tratcher. You are right. I use kestrel without IIS, no issue.  @Tratcher

HttpSysServer may also work for you as it uses the raw url to work around the unescaping caused by Http.Sys. In the IIS+Kestrel scenario this isn't possible today because the raw url is lost when proxying to Kestrel.

Thanks for the response! @rynowak , everyone.
For compatibility reasons. I want to migrate existing project to ASP. NET Core 2.0.
I don't use querystring because I have been using .net framework/iis to do the same thing, and many existing projects depend on it.

@Tratcher where should this bug ultimately go? I just want to make sure it's in the right repo, even if we end up just closing it.

Close, this is an IIS issue. We've dealt with it the best we can in the individual servers.


Unable to use most Knex types with my TypeScript code due to their not being exported

Environment
Knex version: 0.95.0
Updating from: 0.21.12, where the code shown later did work.
Bug

Explain what kind of behaviour you are getting and how you think it should do

The types in typings should be exported so they can be used in applying types to code using Knex.

Error message

A config builder function declared like so:
const knexConfig = (
  {
    RDS_DB_NAME,
    RDS_USERNAME,
    RDS_PASSWORD,
    RDS_HOSTNAME,
    RDS_PORT,
    source,
  } = configFromEnv()
): Knex.Config<Knex.MsSqlConnectionConfig> => ({
// …config options here…
})

Fails to compile with tsc due to Knex.Config and Knex.MsSqlConnectionConfig not being exported:
TSError: knexfile.ts:50:9 - error TS2694: Namespace '"/foo/node_modules/knex/types/index".Knex' has no exported member 'Config'.

50 ): Knex.Config<Knex.MsSqlConnectionConfig> => ({
           ~~~~~~
knexfile.ts:50:21 - error TS2694: Namespace '"/foo/node_modules/knex/types/index".Knex' has no exported member 'MsSqlConnectionConfig'.

50 ): Knex.Config<Knex.MsSqlConnectionConfig> => ({
                       ~~~~~~~~~~~~~~~~~~~~~

I attempted a workaround, but then run into issue reassuring the knex function that the argument is the expected type, since I can't as Knex.Config either:
TSError: knexfile.ts:91:40 - error TS2694: Namespace '"foo/node_modules/knex/types/index".Knex' has no exported member 'Config'.

91   const knex = knexCtor(config as Knex.Config)
                                          ~~~~~~

Reduced test code, for example in https://npm.runkit.com/knex or if it needs real
database connection to MySQL or PostgreSQL, then single file example which initializes
needed data and demonstrates the problem.

This should do it. I couldn't find an online runner for TypeScript that'd let me import a module, but drop this in a TS project:
import knex, { Knex } from 'knex'

const config: Knex.Config = {}

It will error with something like:
TSError: knexfile.ts:9:20 - error TS2694: Namespace '"foo/node_modules/knex/types/index".Knex' has no exported member 'Config'.

9 const config: Knex.Config = {}
                     ~~~~~~

I expect the fix is basically to run through the typings files and stick export in front of a lot of interface and type decls, but the details of TypeScript typings are not my strong suit, so there might be a less verbose way.
I think 177938afeb2abf7d952f4a16b5a7cde213ab0c17 is where the exports changed.
(Ping @lorefnon for typings.)

Taking a look now!

@jeremy-w Which TypeScript version are you using?

Version 4.0.5

See UPGRADING.md for migration guide - TypeScript version 4.1+ is needed. I think that should resolve your problem!

I'll give that a shot and let you know. :)

Thanks!

That seems to have done it. Past the typings issue to the tedious issue.

@jeremy-w If it's currently impossible to pass the necessary param to Tedious through knex, please raise an issue for us!

Must be something else than that config flag - using tedious directly connects fine, but i'm hitting the error as described there - same error, different cause, presumably. If it boils down to a knex issue, i'll let you know. Thanks for your help!

happy to help! if you manage to resolve the issue, please post here too, most likely other users will encounter same problem.

@kibertoad OK, I figured it out. I'll need to file a bug and an enhancement request.
BUG: duplicate connect: use on('connect') not connect()
First off: The TDS_VERSION was a red herring. The version being used was fine, which my test directly using tedious confirmed.
The issue instead is an illegal state operation. The connection failure is due to knex triggering connect when a connect is already in flight. tedious begins connecting the moment its Connection is constructed. knex then calls connect() again; this doesn't trip a state machine transition, so the event that arrives when the second connection succeeds catches tedious unawares, and it blows up. This is visible in this debug logging; the tell is that "TDS-DEBUG: connected to localhost:1433" shows up twice, the second time being just before everything goes boom:
> DEBUG='knex:*' node build/t_knex.js
  knex:mssql connection::connection new connection requested +0ms
  knex:mssql XXX: OK, calling connect now! +11ms
TDS-DEBUG: connected to localhost:1433
TDS-DEBUG:   PreLogin - version:<IP_ADDRESS> 1, encryption:0x02(NOT_SUP), instopt:0x00, threadId:0x00000000, mars:0x00(OFF)
TDS-DEBUG: State change: Connecting -> SentPrelogin
TDS-DEBUG:
TDS-DEBUG: Sent
TDS-DEBUG:   type:0x12(PRELOGIN), status:0x01(EOM), length:0x0035, spid:0x0000, packetId:0x01, window:0x00
TDS-DEBUG:   0000  00001F00 06010025 00010200 26000103 00270004 04002B00 0106002C 0001FF00  .......%....&... .'....+. ...,....
  0020  00000100 01020000 00000000 01  ........ .....
TDS-DEBUG: connected to localhost:1433
events.js:291
      throw er; // Unhandled 'error' event
      ^

Error: No event 'socketConnect' in state 'SentPrelogin'
    at Connection.dispatchEvent (REDACTED/node_modules/tedious/lib/connection.js:1282:26)
    at Connection.socketConnect (REDACTED/node_modules/tedious/lib/connection.js:1304:10)
    at REDACTED/node_modules/tedious/lib/connection.js:1146:12
    at Socket.onConnect (REDACTED/node_modules/tedious/lib/connector.js:106:7)
    at Socket.emit (events.js:314:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1131:10)
Emitted 'error' event on Connection instance at:
    at Connection.dispatchEvent (REDACTED/node_modules/tedious/lib/connection.js:1282:12)
    at Connection.socketConnect (REDACTED/node_modules/tedious/lib/connection.js:1304:10)
    [... lines matching original stack trace ...]
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1131:10)

The anticipated fix is not to call connection.connect, but instead to register an on listener for the connect event, as shown in the first JS sample at https://www.microsoft.com/en-us/sql-server/developer-get-started/node/mac/step/2.html.
ER: emit tedious debug output when in debug mode
knex already interprets the debug: true knex config into tedious's debug: { turn on all the things } config option. but it then doesn't actually register an on('debug') handler, so none of the debugging output requested is ever output!
the debug output you see in my logs above came from me dropping this one-liner into acquireRawConnection():
      connection.on('debug', (msg) => {console.log('TDS-DEBUG:', msg)})

most likely we'd want to use debug rather than console.log, but you get the idea. ;)

All that sorting done, I actually won't be able to adopt the newer tedious-based knex till tedious supports using UTF-8 collations with varchar, as tracked by https://github.com/tediousjs/tedious/issues/1143 With the older driver, I was able to register a mapping from string to varchar rather than nvarchar trivially:
knex.client.driver.map.register(String, mssql.VarChar) // Default `string` type to `varchar`

apparently, this is not only not trivial with tedious, it's entirely unsupported as of now. >.<

@tylerjwatson Could you take a look at last comment?

Holy moly, there is a lot of stuff in that comment.

Am I right in assuming there's a connection issue in mssql?  I haven't been able to reproduce anything of that sort.

I am not familiar with driver.map and haven't implemented anything of that sort in mssql tedious, can I please get some help outlining that mechanism?

The debug thing I can do.

@jeremy-w are you able to provide a minimal reproducible sample so I can test what I am fixing is working?

Upon further investigation, Microsoft's own sample at https://www.microsoft.com/en-us/sql-server/developer-get-started/node/mac/step/2.html doesn't work.  It's not possible to avoid calling .connect() in tedious, and that pattern is used everywhere in tedious' own samples.

Hmm. I ran that first sample, and it connected successfully. Which is a useful tell.
I now suspect I somehow wound up with an older Tedious package: There was a breaking change on 13 January https://github.com/tediousjs/tedious/commit/16b10bf0f01d998de3aadfac2e2794d20aed4775#diff-a6be0583428e46844273df76939f02077073da3075716fc57d291a5f2463eaf5 that moved from autoconnect on construction to waiting for the caller to initiate the connection.
I’m curious how that happened with version constraints, and if they maybe need bumping to prevent that.
I’ll have a look when I next can. (Probably by Wednesday at latest, but I might get a few minutes to nose around this weekend.)
Thanks for your help!
Jeremy W. Sherman
http://jeremywsherman.com/

On Mar 4, 2021, at 18:16, Tyler Watson<EMAIL_ADDRESS>wrote:

Upon further investigation, Microsoft's own sample at https://www.microsoft.com/en-us/sql-server/developer-get-started/node/mac/step/2.html doesn't work. It's not possible to avoid calling .connect() in tedious, and that pattern is used everywhere in tedious' own samples.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Confirmed Knex 0.95.0 has a peer dep constraint of ^11.0.3 https://github.com/knex/knex/blob/0.95.0/package.json#L60 and the Tedious breaking change happened in v10.0.0 https://github.com/tediousjs/tedious/releases/tag/v10.0.0 so I’m at a loss for how I wound up with new Knex and old Tedious installed without npm yelling at me. But that appears to be the root of my issue.
I hope this helps anyone else running into this issue.

@jeremy-w Perfect! I'm glad the fix was a simple one.

That’s what’s odd: It was installed (I ran the Microsoft sample code), but it was old. Missing would have been obvious; version-incompatible was a total surprise to me!


Can't find captcha img tag

Hello,
Today there seems to be a major issue that has broken seroposcope.
[2022-01-29 01:05:58,004] [google-2] DEBUG c.s.s.s.g.s.GoogleScraper - captcha form detected via proxy:http://xx.xxx.xxx.xx:3128/
[2022-01-29 01:05:58,164] [google-0] DEBUG c.s.s.s.g.s.GoogleScraper - noscript form detected, trying with captcha image
[2022-01-29 01:05:58,164] [google-0] DEBUG c.s.s.s.g.s.GoogleScraper - can't find captcha img tag
[2022-01-29 01:05:59,730] [google-1] WARN  c.s.s.t.g.GoogleTaskRunnable - scrap failed for XX because of ERROR_NETWORK
All of my proxies are working and active, so I know it's not them. There seems to be some issue with being able to get the captcha key from Google.
Is anyone else experiencing this?

Seems the issue may have been resolved by updating to the latest version... Testing now.


zooming issues

*I can't zoom out as far in DAIL as I am in vanilla - any advice or is something wrong?

Fixed in 54


IPv6 gateway endpoint support

Our company is on a IPv6 readiness process and need explicit confirmation on IPv6 support as an endpoint for the VPN-Gateway on a Site-to-Site configuration. The document is not explicit about this. The document does not mention it as an option so we assume it is not possible, but a warning or alert stating this will make it explicit.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 57cbce6b-e48b-c199-7071-865a2a6422f0
Version Independent ID: 06320049-4747-cd1f-564a-36f3a6f5b004
Content: Connect on-premises network to Azure virtual network: Site-to-Site VPN: Portal - Azure VPN Gateway
Content Source: articles/vpn-gateway/vpn-gateway-howto-site-to-site-resource-manager-portal.md
Service: vpn-gateway
GitHub Login: @cherylmc
Microsoft Alias: cherylmc

@nextorr Are you referring to an Azure VNET with IPv6 Enabled, or connecting to a VPN Gateway via  a Public IP with IPv6 enabled?
Currently, Azure VPN Gateways (A required prerequisite for a S2S VPN) are not compatible with an IPv6 VNET.

Hello, im referring to the later, connecting to a VPN Gateway via a Public IP with IPv6 enabled.  The document you are linking adds to the confusion, as the document i am commenting on states that i can add an IPv6 range to the VNET associated with the Gateway: "IPv6: If your configuration requires IPv6 address space, select the Add IPv6 address space box to enter that information." now it is less clear if i can or cannot use IPv6.

@nextorr At this time, you cannot use a Public IP with IPv6 for a VPN Gateway. This feature is currently planned, but I am unable to give a timetable for its release. If you would like us to prioritize this feature, please contribute to this existing piece of feedback.
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


Persistence / Testnet

I wanted to see if there's a way to implement persistence pretty quickly, given that this seems to be the one major thing that's missing before this could be used as a functional MVP? Looking at the docs and the code, seems that everything's ready to go for generating wallets, doing transfers, etc? If there's more work needed before multiple nodes could communicate in a testnet type environment, let me know and I'll get to work.

To be 100% honest there is more than just persistence missing right now. Block propagation and validation is still very rudimentary and untested.
Then there is some logic missing like chain splits and the most POW rule plus many of the block BIPs.
Anyway I'd like to get to that stage where we can start slowly introducing persistence, starting with blocks. Personally I'd like the format to be compatible with Core.
Not sure if just dumping each block's binary data onto its own file is enough? Not sure what happens when reverting to another chain... I know core has some rev files as well for blocks on disk.
Also I'd like to investigate using actors for the persistence layer somehow to increase data race safe parallelism.
Then there is the elephant in the room that is the database. This will certainly be needed for chain state and TX index. I'm trying to keep Swift Bitcoin free of 3rd party deps (aside from SwiftLang/Apple and secp256k1 ).
I could eventually vouch for leveldb but even that seems like an overkill. Foundation's UserDefaults currently does not work on linux. I'm leaning towards implementing our very own efficient key-value blockchain database ala hammersbald, the DB from the rust-bitcoin project.
In either case, I think we're not quite there yet but if you'd like to experiment possible solution on the side. I will focus on getting all the block algos and tests in place so we can be in a good position to persist.


Update Readme to include AspNet SignalR support

Rendered view

One comment: the first sentence we would like it to be more specific: "Azure SignalR Service SDK for .NET helps you build ASP.NET based Azure web application with ASP.NET Core based Azure SignalR Service."
Others look good.


fix: suggest alternative argument for well-known unknown argument

What does this PR try to resolve?
Part of #11702.
This pull request replaces manual errors with clap native UnknownArgumentValueParser::suggest to give uses alternative arugments, including

Suggest --no-fail-fast for --keep-going in cargo test and cargo bench.
Suggest cargo-vendor vendor for unsupported cargo tree arguments

--no-merge-sources
--relative-path
--only-git-deps
--disallow-duplicates

This PR also renames fn arg_jobs(…) and friends to more reasonable names.
How should we test and review this PR?
cargo t --keep-going
cargo vendor --no-merge-sources
cargo vendor --relative-path
cargo vendor --only-git-deps
cargo vendor --disallow-duplicates

Before
$ cargo vendor --no-merge-sources
error: the crates.io `cargo vendor` command has now been merged into Cargo itself
and does not support the flag `--no-merge-sources` currently; to continue using the flag you
can execute `cargo-vendor vendor ...`, and if you would like to see this flag
supported in Cargo itself please feel free to file an issue at
https://github.com/rust-lang/cargo/issues/new

After
$ cargo vendor --no-merge-sources
error: unexpected argument '--no-merge-sources' found

  tip: the crates.io `cargo vendor` command has been merged into Cargo
  tip: and the flag `--no-merge-sources` isn't supported currently
  tip: to continue using the flag, execute `cargo-vendor vendor ...`
  tip: to suggest this flag supported in Cargo, file an issue at <https://github.com/rust-lang/cargo/issues/new>

Usage: cargo vendor [OPTIONS] [path]

For more information, try '--help'.

This is current blocked on https://github.com/clap-rs/clap/issues/5081.

This is current blocked on clap-rs/clap#5081.

This can be worked around on the clap side.  We could add to the prelude a maybe_flag function that calls try_get_one and filters out the type-discrepancy error.

@epage this is ready for review. Moved some functional tests to UI tests since it makes more sense matching the entire clap output there.

@bors r+

:pushpin: Commit a31aaac5367b28d1bfe9c6debb0ad658dab73e60 has been approved by epage
It is now in the queue for this repository.

:hourglass: Testing commit a31aaac5367b28d1bfe9c6debb0ad658dab73e60 with merge a4c5f1127fcf33a5d15982316bc55a92d50462c3...

:broken_heart: Test failed - checks-actions

@bors r+

:pushpin: Commit 77da5145294f5fe5fda3fcbd0ad336d1fb0cd471 has been approved by epage
It is now in the queue for this repository.

:hourglass: Testing commit 77da5145294f5fe5fda3fcbd0ad336d1fb0cd471 with merge 50ba4bfa0cf536a1d0cfbe0840800a098a1ee023...

:sunny: Test successful - checks-actions
Approved by: epage
Pushing 50ba4bfa0cf536a1d0cfbe0840800a098a1ee023 to master...

:sunny: Test successful - checks-actions
Approved by: epage
Pushing 50ba4bfa0cf536a1d0cfbe0840800a098a1ee023 to master...

:eyes: Test was successful, but fast-forwarding failed: 422 Changes must be made through a pull request.


Docs and cleanup

Just want to merge this small amount of progress before branching off onto more substantial work.


Separate Languages into Individual Files

Just a thought... It might make it easier the make separate files for each language. Also, you could possibly turn this into a website using https://pages.github.com/ as a host.
This is just my 2 cents/pence/yen.
Feel free to close this issue without comment if you are not interested.

I'll do it as soon as possible. Thank you!

@jpyper It's done! da3e252


HtmlStapler breaks HTML or takes very long to load

Im having very strange behaviour with HtmlStapler. When using it with default setup with a very simple HTML with no JS nor CSS, the webpage takes about 20 seconds to load. When it includes a simple local CSS, the HTML returned is broken (it doesnt return the whole file, resulting in stripped content and an invalid HTML).
I'm using Jodd 3.6.4, but I tried it with previous versions with no luck.
I've reproduced this situation in:

Windows 7 64-bits, Tomcat 8.0.20, JDK 8u40 and JDK 7u76
Windows XP 32-bits, Tomcat 8.0.12, JDK 8

This error does NOT happen when deploying in Linux (Centos 7 64-bits, Tomcat 8.0.20, JDK 8u40), it works just fine.
Ive created a very small webapp where you can check this issue. It contains 3 HTML with these scenarios:

Compiled WAR: https://dl.dropboxusercontent.com/u/159166/jodd-demos.war
Source code: https://dl.dropboxusercontent.com/u/159166/jodd-demos.zip

Wow, that is something weird going on! It looks this is Windows related; Im installing the windows box at the moment and will try to figure what is going on.
Thank you for notifying; this type of issues are especially hard to detect :) Stay tuned!

Would you be so kind to test on Tomcat 7 by any chance? I now remembered that we experienced weird issue with filters and Tomcat 8 that sounds like yours: HTML page is stripped and long response. This test was on Mac, however; still; it might be something with Tomcat version, as version 7 worked for us.

Ok, I've tested this on Tomcat 7.0.59 with JDK 7u76 and JDK 8u40: the file test1.html loads quickly with no problems. The file test2.html takes 20 seconds to load. The file test3.html loads quickly with no problems.
This still looks like Windows related, but with Tomcat 7 there is a slight different behaviour.

Ok, thanx, will investigate this (and call ghostbusters :)

It seems there are two overlapping issues. I am now inspecting the issue with Tomcat 8, that I can reproduce;  and it is quite weird one. Stay tuned, working on this...

Great, let me know if there is anything I can do to help (maybe calling a voodoo witch or something...)

OK, figured out something: Tomcat 8 realizes that e.g. test3.html is a resource, i.e. a file, and sets the content length to file size; while real file size of modified file is greater then that.

OK, it seems that reseting content length is solving the issue (at least regarding tomcat8, will check windows tomorrow).
Meanwhile - would it be possible for you to test the new jars? I will commit the fix; so you can either:

checkout github
i can send new jars via email
i can publish a snapshot to snapshot repo (which is not maven central, but nexus one)

Just let me know what works best for you!

If you can just send me the jars thats best.

The fix is committed. I have the jars. I just need your email :)))
Can you send me your email? Here is how you can contact us:
http://jodd.org/contact.html
Thanx for that voodoo, it helped ;) Let's hope this solves issue on Windows, as well; didn't have time to check it yet :(

Thank you @ldeseta for testing, everything works now, so I will close the issue.
p.s. also take a look at the 92163dafdcc1ff225ddf732a69e821e53faa4634 :)


Allow changing the daemon log level

Current Behavior
I am classifying this as a bug after discovering a 100GB folder of daemon logs, all at the debug level. Left unchecked, this could continued to grow until it destroys my hard drive.
The Gradle Daemon is hard-coded to log at the DEBUG level between builds and completely ignores the user-set log level. There is an open issue to change the default log level of the daemon. That issue has been open for years. A maybe simpler and more pressing issue is that at the users at least have the ability to change the daemon log level.
I made a new issue given that perhaps there is some blocker making it complicated to change the default log level. Maybe that issue is difficult to implement quickly because it will immediately affect all users, and so there may be a lot of thought that has to go into it. This issue, however, would be opt-in and not change the default behavior for anyone. Hopefully that would reduce the concerns that is perhaps blocking the other issue.
Expected Behavior
There should be a property that allows us to change the daemon log level.
Context (optional)
No response
Steps to Reproduce

Check the ~/.gradle folder and see large log files with debug logs

Gradle version
8.3
Build scan URL (optional)
No response
Your Environment (optional)
No response

Thank you for your interest in Gradle!
This feature request is in the backlog of the relevant team, but this area of Gradle is currently not in focus. It might take a while before it gets implemented.


Inline code with angle brackets nested in JSX tag causes highlighting to fail

Subject of the issue
If I have a program like this:
<Hey>
`<F<` x
y
</Hey>

This has weird syntax highlighting:

This case is minimal. If I remove e.g. the F or an angle bracket <, then the highlighting returns to normal:

Your environment

OS: MacOS
Packages: n/a
Env: VSCode 1.61.2, VSCode MDX 0.2.3

Another example of this:

Code:
<RevealAnswer>

abc `a <firstName>`

</RevealAnswer>

error`Unexpected token, expected "{"`

What's the problem here?

```javascript showLineNumbers
function addAndDivideByFive(numberA, numberB)
  console.log('numberA', numberA) {
  const sum = numberA + numberB;
  const result = sum / 5;
  return result;
}
```

Solution: We broke the function statement because we misplaced the `console.log`. The error message already gave us a hint about that. A `{` is expected right after specifying the parameters because the function


OS flag never cleared in RV-8263-C8 driver

Describe the bug
The flag is never cleared and so the oscillator doesn´t run anymore if the flag is set. The flag must be cleared by software.
Notes
@jakubtopic
We missed this in our initial discussion about the OS flag. There is an issue that get_time doesn´t work anymore as soon as the flag is set. I would like solve this issue in this way in rv8263c8_time_get:
	/* Return an error when the oscillator is stopped. */
	if (regs[0] & RV8263_BM_OS) {
		/* Clear the OS flag. */
		regs[0] &= ~RV8263_BM_OS;
		i2c_reg_write_byte_dt(&config->i2c_bus, RV8263C8_REGISTER_SECONDS, &regs[0]);
		return -ENODATA;
	}

So the application get´s an error when the flag is set and the flag is reset. If this error happens again the application will get the same error again.
What do you think? I think it´s the best way to handle it because you have to clear it by software every time.

Please use our bug template when reporting bugs. You need to edit this issue to include the information requested in https://github.com/zephyrproject-rtos/zephyr/blob/main/.github/ISSUE_TEMPLATE/001_bug_report.md


Debugger File Selection Improvements

Load files using canonical names. (Resolve symlinks so that IDEA does not open
a symlink as a duplicate buffer.)
Use implicit class path entries when searching for relative paths. (haxe/std)

@sganapavarapu1 Please review.

CI builds are failing

The CI builds are failing on 14.1.1 because the JetBrains site isn't serving the IDEA release for 14.1.1 at the moment.  It has nothing to do with the sources.  Please continue the review.  Thanks.

I approve this for merge (assuming you'd change that AS3/JAVA_SCRIPT to AS3/FLASH)

Thanks.


Adds a new function Stop-VSTeamBuild

PR Summary

Adds a new function Stop-VSTeamBuild which allows cancelling a build using the build id.
Closes #206

PR Checklist

[X] Write Help
[X] Write Unit Test
[X] Update CHANGELOG.md

Hi @brittandeyoung
looks great already. One small thing is missing. Could you create a help document like the following?https://github.com/DarqueWarrior/vsteam/blob/master/.docs/Add-VSTeamBuild.md
A cmdlet should have help page which links the synopsis. Thank you!

Hi @brittandeyoung
looks great already. One small thing is missing. Could you create a help document like the following?https://github.com/DarqueWarrior/vsteam/blob/master/.docs/Add-VSTeamBuild.md
A cmdlet should have help page which links the synopsis. Thank you!

I have added the requested help page. Please let me know if there is anything else I am missing.

I merged the PR and adjusted a few lines of code for optimization and added a unit test for testing the pipeline feature for build IDs.
Thanks for the good and quick contribution!


Create firmware support for input detection

This PR has no description. Should it be a draft?

This PR has no description. Should it be a draft?

My bad, I chose the wrong fork to make the PR to, this should be to my personal branch 'EuroPi-X'. I'll close and move


Support authentication and server-side user state

There are quite a few features (#9, #10, #11) that require some kind of state. In those feature descriptions, its mentioned to use client-side storage for the time being. This feature request is to track storing that information server-side and adding authentication support.
The goal right now is to keep it as simple as possible, while also providing enough flexibility for the current and near-term needs.
If you have any thoughts on this, or a better proposal, please speak up!
Here's the gist of it:
Automaat Server

Add a users table to the database, containing the following columns:

api_token – a personal token used to authenticate
preferences – a JSON-based column to store preferences

Add a CLI argument (e.g. --enable-authentication) to require API requests to contain a header with a user token.
Add a new GraphQL query to get the preferences of a user.
Add a new GraphQL mutation to allow resetting the API token, and also changing the stored preferences.

Automaat Web Client

When receiving a 401 response, show a full-screen dialogue asking the visitor to provide a valid user token (which they would get by asking a server admin to provide them one).
Store this token in a secure cookie for future use.
Add this token to all requests.
On startup, fetch user preferences from server using the server API

Notably missing from this proposal is creating user accounts. This will have to be done manually by server admins by adding a record to the database using a raw SQL insert statement. This keeps things simple, and allows us in the future to add an optional --allow-user-registration server flag to start exposing this functionality (if there is ever a need).

This implementation is not the meant for public access security. Given that Automaat is a powerful tool if exposed in your organisations infrastructure, I think we should explicitly mention that this tool is meant to run inside your internal network (or exposed over a VPN), the authentication added here is merely meant to:

Allow a specific set of people within your organisation to access its functionality.
Persist user preferences across clients/devices.
Provide future-proofing for when we add role-based access models to further segregate who gets to do what.

Another thing we can/should do once this lands:
Add weak references between jobs[1] and users, so that you can view a history of the tasks you ran in the past, and re-view their output.
If we're doing this, we might later add some kind of way to "archive" jobs, so that they are no longer visible in the history, and perhaps automate this for older jobs, for security/privacy related purposes.
1: (which are currently named tasks, but there a plan to rename pipelines to tasks, and tasks to jobs)

An excellent point by @jewilmeer:
The reference between jobs and users could not only serve as a way to view your job history, but also (more importantly) as an auditing trail to see who ran what task.

Initial server-side support has been implemented in 411e8e21037fd1bb9ea1ee94de7920b25ae8c4f0.

Another step towards solving this issue has been implemented in 59166f4a09372853ac43dfbb83031dcc04784e40.
With this change, the client<>server authentication loop is implemented. The next step is to use the new fundamentals to start storing persistent session state on the server.
Here's a screenshot of the new login page:

This has been implemented in bf9ffe4cd9e148d3ecc0322e0c3bb87eca907d5f.


Fixed #6

Hey, Everything is good.
There are some changes though :

Can you do this with class instead of struct?
Write ```cpp instead of only 3 backticks

I'll try with class.

Open PR again when done!


Set initial selection to ALL when the page first loads

If there is no current filter, then turn on the all filter. e.g., after remove 4

No remove button for ALL

ALL can not co-exist with other filters


Fix issue with times being generated in the DST gap

Increased the number of ScalaCheck tests to expose this issue, too.
@franciscodr @raulraja @diesalbla would you be able to take a look please?
Fixes #15.

Hold off the PR for now... looks like increasing the tests on Travis has uncovered another issue.

Current coverage is 100% (diff: 100%)

Merging #26 into master will not change coverage

@@           master   #26   diff @@
===================================
  Files           8     8
  Lines          78    81     +3
  Methods        78    81     +3
  Messages        0     0
  Branches        0     0
===================================
+ Hits           78    81     +3
  Misses          0     0
  Partials        0     0

Powered by Codecov. Last update 50b0922...afea55b

LGTM!


[SPARK-48505][CORE] Simplify the implementation of Utils#isG1GC

What changes were proposed in this pull request?
This PR changes to use the result of ManagementFactory.getGarbageCollectorMXBeans to determine whether G1GC is used. When G1GC is used, ManagementFactory.getGarbageCollectorMXBeans will return two instances of GarbageCollectorExtImpl, their names are G1 Young Generation and G1 Old Generation respectively.
Why are the changes needed?
Simplify the implementation.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GitHub Actions
Was this patch authored or co-authored using generative AI tooling?
No

also cc @yaooqinn FYI

It's not my area. So basically, I rely on the question below to be properly answered.

How was this patch tested?

Can we point out the test case that covers it?

@yaooqinn Although the logic is a bit convoluted, I am confident that the following two cases cover the scenarios where isG1GC is true and false, respectively.
https://github.com/apache/spark/blob/8d9d9c243ed141660e853b189982979d19f0ba9d/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala#L145-L159
These two cases executed org.apache.spark.examples.DriverSubmissionTest, intercepted the logs, and checked the return value of Utils.isG1GC in the logs.
https://github.com/apache/spark/blob/8d9d9c243ed141660e853b189982979d19f0ba9d/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala#L306-L343
https://github.com/apache/spark/blob/8d9d9c243ed141660e853b189982979d19f0ba9d/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala#L656
https://github.com/apache/spark/blob/8d9d9c243ed141660e853b189982979d19f0ba9d/examples/src/main/scala/org/apache/spark/examples/DriverSubmissionTest.scala#L46
and they passed

Thank you @LuciferYang , merged to master

Thanks @yaooqinn

Thanks for the ping early on. I was holding back on making a comment to avoid derailing the thread, but thought I'd share some thoughts still.
I'm neutral to this change. It's a change that goes from one implementation-dependent code to another one of a different shape, the net-net is that both works in practice but it's hard to argue which one is superior. For the sake of having less code and avoiding reflection, I'm okay with the newly merged code, but I'd like to make sure we understand the caveats.

the old code

Cons: uses reflection to probe the existence of the HotSpot VM specific HotSpotDiagnosticMXBean; but it's not too bad, at least it doesn't require setAccessible(true)...
Pros: the -XX:+UseG1GC flag is precise and there's no room for any ambiguity

the new code

Pros: plain Java code, no reflection
Cons: there's no hard contract/guarantee that this will match and will only match G1 GC; it's not too bad in practice, that at least for now it'll work just fine on current commonly used JVMs

The string matching reminds me of some test code that used to be in the JDK that matched the exact exception string from certain exceptions (which wasn't a hard contract that should have been depended on), and when the JVM side made some optimizations that changed the string's spelling, that test was broken. You can argue either way.
The old code's initial attempt to use the HotSpotDiagnosticMXBean didn't use reflection, but rather it used the APIs directly, and later changed from direct use to reflection-based used to allow it to run on all JVMs. In fact, the very first commit in the original PR had pretty much the exact same code as the new one proposed here 😓
https://github.com/apache/spark/pull/34846/commits/3a4c66117b157a8a73cc32deab0dc4e8c632db38#diff-2ecc6aef4b0c50bbf146e6c0b3b8b2249375f06a83e2a224c7718cfc850c3af7R3238
It's already very platform dependent when you're trying to do G1GC detection; the use of HotSpotDiagnosticMXBean falls in the category of "Using Oracle JDK's Platform Extension" (JDK doc, JDK 17's doc also still links back to this JDK 8 page). I'd say I'm okay with the old reflection-based version for its preciseness.
It was using reflection all the way, but it's also possible to only do an initial reflection probe and then conditionally do direct use of HotSpotDiagnosticMXBean so that it "looks" less bloated.

Thanks for the ping early on. I was holding back on making a comment to avoid derailing the thread, but thought I'd share some thoughts still.
I'm neutral to this change. It's a change that goes from one implementation-dependent code to another one of a different shape, the net-net is that both works in practice but it's hard to argue which one is superior. For the sake of having less code and avoiding reflection, I'm okay with the newly merged code, but I'd like to make sure we understand the caveats.

the old code

Cons: uses reflection to probe the existence of the HotSpot VM specific HotSpotDiagnosticMXBean; but it's not too bad, at least it doesn't require setAccessible(true)...
Pros: the -XX:+UseG1GC flag is precise and there's no room for any ambiguity

the new code

Pros: plain Java code, no reflection
Cons: there's no hard contract/guarantee that this will match and will only match G1 GC; it's not too bad in practice, that at least for now it'll work just fine on current commonly used JVMs

The string matching reminds me of some test code that used to be in the JDK that matched the exact exception string from certain exceptions (which wasn't a hard contract that should have been depended on), and when the JVM side made some optimizations that changed the string's spelling, that test was broken. You can argue either way.
The old code's initial attempt to use the HotSpotDiagnosticMXBean didn't use reflection, but rather it used the APIs directly, and later changed from direct use to reflection-based used to allow it to run on all JVMs. In fact, the very first commit in the original PR had pretty much the exact same code as the new one proposed here 😓 3a4c661#diff-2ecc6aef4b0c50bbf146e6c0b3b8b2249375f06a83e2a224c7718cfc850c3af7R3238
It's already very platform dependent when you're trying to do G1GC detection; the use of HotSpotDiagnosticMXBean falls in the category of "Using Oracle JDK's Platform Extension" (JDK doc, JDK 17's doc also still links back to this JDK 8 page). I'd say I'm okay with the old reflection-based version for its preciseness. It was using reflection all the way, but it's also possible to only do an initial reflection probe and then conditionally do direct use of HotSpotDiagnosticMXBean so that it "looks" less bloated.

@rednaxelafx Thank you very much for your explanation, based on your explanation, I think we should roll back this pr.
WDYT @yaooqinn @beliefer ?

It was using reflection all the way, but it's also possible to only do an initial reflection probe and then conditionally do direct use of HotSpotDiagnosticMXBean so that it "looks" less bloated.

@rednaxelafx I've submitted a PR to try and make further refactor according to this suggestion. If you have time, could you please take a look?

https://github.com/apache/spark/pull/46873/files

Thank you @rednaxelafx for the detailed inputs.
I reverted this with https://github.com/apache/spark/commit/db527ac346f2f6f6dbddefe292a24848d1120172, since it's controversial
Let's discuss further in https://github.com/apache/spark/pull/46873 @LuciferYang

Thanks @yaooqinn , I reopened SPARK-48505


Fix 404 errors due to gh-pages

Fixed the issue where 404s would be returned when navigating to site
This issue was due to gh-pages not playing nicely with SPAs
https://create-react-app.dev/docs/deployment/#notes-on-client-side-routing
Fixed by adding a 404 page with some redirect magic as explained on above link
Reference tweet:
https://twitter.com/osquery/status/1397619077707964418

I tested this locally and it all of the functionally still works, happy to merge and deploy.

🙌 Thanks everybody, and nice work @willrushi!


fix typos in README

fix typos and change a few words

Thank you very much for your help! I’ll pick up this change and integrate it with the others.


Rely on Pipfile.lock instead of requirements.txt

Fix #209
Remove Pipfile.lock from .dockerignore, since the install in Docker will depend on it. (Ignored files can not be added to the image.)
Bump base image to python 3.10 to be consistent with Pipfile.lock.
Remove requirements.txt and references to it in the docs.
Split COPY . ./ into separate COPY statements, so rebuilds that touch the code but not dependencies will be quicker.

I've built and run this locally, and it works: One possible next step is to do the docker build in CI, and point Cypress at that.

I also notice this warning when I run the docker image:
docker build --tag clima .
docker run clima
...
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.

I haven't used gcloud, but it looks like it can work either with a docker image (what clima is doing, I believe), or with python source code. I'm a little out of my depth here, but even though it's running inside a docker container, it is still just running a development server. My guess is that with a WSGI server performance might be a little better, or the google bill might be a little smaller.
If this is worth worrying about, it might be simplest to drop our Dockerfile, and use the gcloud python source code option. One constraint there is that they do not support Pipfile. Another option is pip-tools : essentially keep your high-level dependencies in requirements.in, and compile a requirements.txt that tracks dependencies.

@mccalluc thank you again for your great work.
May you open a separate issue about this:

I also notice this warning when I run the docker image:
docker build --tag clima .
docker run clima
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
I haven't used gcloud, but it looks like it can work either with a docker image (what clima is doing, I believe), or with python source code. I'm a little out of my depth here, but even though it's running inside a docker container, it is still just running a development server. My guess is that with a WSGI server performance might be a little better, or the google bill might be a little smaller.
If this is worth worrying about, it might be simplest to drop our Dockerfile, and use the gcloud python source code option. One constraint there is that they do not support Pipfile. Another option is pip-tools : essentially keep your high-level dependencies in requirements.in, and compile a requirements.txt that tracks dependencies.

Yes, currently we are using a docker image and deploying it using cloud run. I only was aware of this solution, I did not know that we could have deployed the code on Google Cloud run in another way. Please let me know if you have the time to look into this in a bit more depth, maybe we can find a better solution to deploy our applications in the future. One major downside of using Docker images is that they are very slow to start if no one has used the application before.


#552 Fixes a bad config for 0.2.1

What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #552
Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

This change is

/lgtm
/approve


broadcast-agent-discovery - supporting multiple network interfaces

I am working with the broadcast-agent-discover.py script and that is working well, but only with a single network interface.
The network environment I need to support has (at least) two active network interfaces, and I need to discover devices in both.
I have tried variations of the suggestion in this page with no luck: https://pysnmp.readthedocs.io/en/latest/faq/listening-on-multiple-interfaces.html.
When the second network is active the script still finds the devices in the main network, but does not see devices in the second.
Looking for pointers on what needs to be added.
Thanks!

https://pysnmp.readthedocs.io is an invalid site and we don't cover anything it says in this repo.
If your OS/network stack doesn't like <IP_ADDRESS> to broadcast on both networks, you might try to broadcast to either of them with the right broadcast address and then combine the results.


Some smaller updates

Includes a new test for bm25 row-based upsert with schema
Updates the type hint for attributes from Optional[str] to Optional[str | int], we support numerical attribute types

Good catch. Sorry I missed this.

@pushrax went ahead and bumped version to v0.1.13 in here, mind merging this + doing the publish after final review? Added the schema param to the upsert helpers too, I don't think this changed anything, but I agree it's the right thing to do!

published


Nice app, I’m considering integrating this with VesselAlert

Hi there,
I came across your short script/app via the AisCatcher website. Great idea! I’m considering to incorporate it into my vessel notification app/container called VesselAlert , of course with proper attributions under your GPL license. One of the improvements I’m thinking of is to keep a rolling (12-hour for example) max distance and only send a notification if the actual distance is > minimum distance and if it’s at least (say) 25 nm further than the last measured max distance. That way the notifications can be limited to only meaningful ones.
This would be a great add to VesselAlert and easy to implement as I already have Mastodon and Discord notifications. (See for example https://airwaves.social/@bostonharbor )
Thanks again; feel free to close this (non-) issue after you read it.
Ramon

Sounds great, I hope you find the script useful!


Update boosting.html

Added a common sense introduction to Adaboost along with the paper
Added example to the calculation
Added note on outlier
Updated a few formulae as per the paper

One request is merge this with your gh-pages branch, post that edit what you want. Please do not copy the changes manually in your branch. If you copy the changes manually, it creates a conflict with my branch due to which I had to delete my entire repository and re-fork again. We can discuss more if needed.


feat: allow custom authentication component in header (clevercanary/hca-atlas-tracker#395)

See also pull request clevercanary/hca-atlas-tracker#398

Obsolete! Closing!


Uploading a file in a book is failing with a 422 http code

Cannot upload an attachment in a book to attach it into the book
Steps To Reproduce

Go to Books
Add a new book
Select the attachment icon on the right tab
Click on "upload a file" and select it
In the network, you can see a 422 http code ( Unprocessable Entity) is returned with that message {"file":["validation.uploaded"]}
At the same time as 5., in the console you can see :

POST https://rtfm.proxiad-axe-seine.com/attachments/upload 422 (Unprocessable Entity)
value @ app.js?version=v0.25.5:51
value @ app.js?version=v0.25.5:51
(anonymous) @ app.js?version=v0.25.5:51
(anonymous) @ app.js?version=v0.25.5:51
transformFile @ app.js?version=v0.25.5:51
o @ app.js?version=v0.25.5:51
value @ app.js?version=v0.25.5:51
value @ app.js?version=v0.25.5:51
value @ app.js?version=v0.25.5:51
value @ app.js?version=v0.25.5:51
value @ app.js?version=v0.25.5:51
(anonymous) @ app.js?version=v0.25.5:51
setTimeout (async)
value @ app.js?version=v0.25.5:51
(anonymous) @ app.js?version=v0.25.5:51
accept @ app.js?version=v0.25.5:51
value @ app.js?version=v0.25.5:51
value @ app.js?version=v0.25.5:51
(anonymous) @ app.js?version=v0.25.5:51

Expected behavior
I can upload and link a file to my book
Screenshots
Your Configuration (please complete the following information):

Exact BookStack Version (Found in settings): 0.25.5
PHP Version: have to ask
Hosting Method (Nginx/Apache/Docker): have to ask

@nLenouvelProxiad I think this can often occur if the file fails to upload due to server upload size limits or due to file permissions.
I see your BookStack instance has since been updated to the latest version. Give it a try again and let me know if you still experience the same issue, without a helpful error message.

Due to the lack of response from the original issue author I will close this.
If this issue remains please feel free to open a new issue, referencing this one.


Changes to NectarCAMEventSource

Something odd seems to be happening when reading a NectarCAM file for a second time (see the error message that occurs in even the first commit, before any changes to the NectarCAMEventSource class). @dneise  Any idea why this error occurs (it happens inside protozfits/simple.py)

I believe this is connected to an issue I ran into nearly a month ago .. but I was too lazy to even completely describe the issue. I am very sorry.
https://github.com/cta-sst-1m/protozfitsreader/issues/25

Can you merge in the master now and fix the conflicts? I merged #734, so perhaps that will resolve this

A segfault is thrown when one of these events go through ctapipe.utils.neighbour_sum.get_sum_array, likely due to the dual channels. Its probably not worth fixing as there will be only one channel at that level when #683 is merged.


Paper 8

https://github.com/UK-Catalysis-Hub/XAS-Workflow-Demo/blob/main/psdi_phase_1/larch/Paper 08 Reproduce XAS.ipynb

Reproduction of the EXAFS fits in this paper requires generating FEFF paths for PdO and Pd metal via the Larch FEFF tool. @subindev-d and I found an issue. Using the tool with a cif file for PdO works fine, but using the tool with a Pd metal cif file throws an error. (I obtained both cif files from the CCDC). This looks like the same issue as @alex-belozerov raised for paper 5, so I elaborate there: https://github.com/orgs/MaterialsGalaxy/projects/1/views/1?pane=issue&itemId=49229893.
This is a problem because without the Pd metal path data the Larch Artemis tool cannot be used to perform the fitting in a similar manner to in the paper.

In trying to reproduce the paper I've been following Abraham's notebook (https://github.com/UK-Catalysis-Hub/XAS-Workflow-Demo/blob/main/psdi_phase_1/larch/Paper 08 Reproduce XAS.ipynb), which has proved useful.
Interestingly Abraham does not use any Pd metal data to perform the fit. If I understand what he did correctly he used PdO paths only in his fitting. Perhaps he is aware of the problem with larch for Pd metal, and only used PdO paths for that reason?
It's also worth mentioning that Abraham's repository gives the GDS and SP csv files which he used in the fitting: https://github.com/UK-Catalysis-Hub/XAS-Workflow-Demo/blob/main/psdi_phase_1/larch/pdox_gds_bk.csv and https://github.com/UK-Catalysis-Hub/XAS-Workflow-Demo/blob/main/psdi_phase_1/larch/pdox_sp_bk.csv. I'd be interested to know where these came from. Moreover, the parameters in these csv files are different enough from the defaults in the Larch Artemis tool that the defaults give fits nothing like what Abraham gets using these csv files. It is, however, possible to mimic what Abraham did by modifying his csv files slightly so that the file paths in the csv files are what Galaxy expects. Here are such files which worked for me:
pdox_sp_bk_TU.csv
pdox_gds_bk_TU.csv
Hence I'm currently confident it is possible to reproduce what Abraham did. However I'm still wondering if what Abraham did is the same as what was done in the paper, given the lack of Pd metal paths?


False positive on missing whitespace for React .tsx files

Expected Behavior
No errors raised by tslint.
Actual Behavior
I'm receiving missing whitespace (whitespace) errors for all opening brackets (<).

Misc

I'm using tslint-config-airbnb for rules.
Package versions (project and global<EMAIL_ADDRESS><EMAIL_ADDRESS>

Facing the same issue in ts files.

Using custom tslint rules.
tslint version 5.11.0

Please share the text of the code when positing with screenshots, along with the source of your tslint file (preferably as a small example project git repo)

Closing as we need easy steps to reproduce this


Switch to using Results List in Sarif Explorer instead of launching the problems panel

The extension will still add problems to the problems panel. This change is to no longer open the problems panel with the extension, and instead to load and open the Results List as it now has has more functionality then the problems panel provides.

Fix was checked in #136


Temporarily disable USWDS banner

@btylerburton It looks like the assets from USWDS are not making it to the build. I'll have to dig more into it. For now, please merge this to disable the USWDS banner.
Here is what I was seeing:

I merged some fixes that worked on my local but I'll also have to give this some more attention next week.
https://github.com/GSA/sdg-indicators-usa/pull/1100

Will merge this in the meantime.


Multi-lingual voice cloning

Hi, Thanks for the great work :)
I'm interested in multilingual voice cloning and seems like there is valle-x.
https://vallex-demo.github.io/
Is this something Bark can handle (maybe in the future)?

yup should already work. we provide 10 prompts for each language that we support. you can specify them via history_prompt="{lang_code}_speaker_{0-9}"
as for cloning a custom voice we prevent that for now to make sure there is no misuse of the technology

as for cloning a custom voice we prevent that for now to make sure there is no misuse of the technology

that is inevitable.

please tell me how to voice clone my voice.  i will use it for private and not public .  Can it


BUG in "send_file" and "send_from_directory"

L496 in helpers.py
If i use flask as a package instead of a module,  I want to expose a file which is not in package.
checks in send_from_directory will be passed because of rel path.  However, in send_file,  added abs folder, it occur an error.

I've hit this problem too; perhaps I can clarify what's going on.
send_from_directory tests for existence based on the safe_join of directory and filename. So if directory is relative, this is relative to the current working directory of the process -- pretty unreliable. send_file, however, expands relative paths based on current_app.root_path, and does not have any existence check.
Thus, if the CWD and root_path do not match, send_from_directory may prematurely raise a 404 for files that do exist, or fail to do so for files that happen to exist in the CWD but not the root_path. In the second case, you get a 500 internal error when os.path.getsize or getmtime fail. Very inconsistent.
My suggestsions:

If you must do an existence check and raise a 404, do it from send_file where the final path is known. This would fix the CWD issue and give a consistent error behavior for the two functions. Or at least enforce that directory is absolute in send_from_directory.
The above is still subject to race conditions, so preferably don't do any inspection of the file, leaving the content-length and mtime to the code that actually sends it. At least in the case of x-sendfile on Apache it sounds like this should work: "Sets correct cache headers such as Etag and If-Modified-Since as if the file was statically served" (https://tn123.org/mod_xsendfile/). Unfortunately it would bypass the app's error handling in the event of a 404, but that's possible in any case due to the race condition or other problems like permissions.

send_from_directory tests for existence based on the safe_join of directory and filename. So if directory is relative, this is relative to the current working directory of the process -- pretty unreliable.

Did you try latest master?

Also I think the current errorhandling regarding raising NotFound is intentional -- send_file is used in situations where the app dev knows a particular file exists. E.g. it wouldn't help with debugging if the following code raises a 404:
return send_file('robots.txt')
OTOH I see send_from_directory as a function where the filename may originate from user input, therefore it's usually not a 500 if the to-be-sent file doesn't exist.
We should still take a look at the races though -- maybe just catch exceptions from the send_file invocation inside send_from_directory instead of doing own checks.

Oh, and I'd like to document the errorhandling of these two functions too somewhere (if it's not already documented).

I was using 0.10.1. You're right, this was fixed in #921. And the distinction you make between the functions makes sense, though I agree the docs could make it clearer. @baisk, does this fix your issue too?

@untitaker having trouble following this whole thread.  Is this just open because you want some more documentation?  It doesn't look like there's an actual problem.

Agreed. Docs are a nice to have but I guess it isn't actually on any roadmap.

I am unable to host the files using send_from_directory and send_file on linux machines.


Mega update time

Time to dust off the ol' rx-jupyter (I'm using it in the serverextension now).

Codecov Report

Merging #43 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master    #43   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           7      7
  Lines          99     99
=====================================
  Hits           99     99

Time to ship!


Throw errors on invalid types for data module

There have been a bunch of GraphQL errors lately as a result of users hitting the data functions incorrectly.
Also added two new npm scripts for publishing a beta package.

Ohhh no wonder errors were being hit in GraphQL.  Made a typo.  Fixed now.


SCA SBOM report format support

By submitting a PR to this repository, you agree to the terms within the Checkmarx Code of Conduct. Please see the contributing guidelines for how to create and submit a high-quality PR for this repo.
Description
Support to SBOM report format.
Example:
.\cx results show --scan-id 8d8b1581-63aa-4e61-b8e0-c6a98effe77e --report-format sbom --report-sbom-format spdxjson
For more details:
.\cx results show --help
References
https://checkmarx.atlassian.net/browse/AST-23496?atlOrigin=eyJpIjoiODg2NmNmYzVjMzBiNGJjNDk1YzQ2ZDYyMjNiZmUxYWEiLCJwIjoiaiJ9
Testing
unity and integration tests
Checklist

[x] I have added documentation for new/changed functionality in this PR (if applicable).
[x] I have updated the CLI help for new/changed functionality in this PR (if applicable).
[x] All active GitHub checks for tests, formatting, and security are passing
[x] The correct base branch is being used

Did we handle the case where the API is not present / functional?


Postal cluster

We are planning a big postal deployment in a cluster way. The DB server will be separated from the app servers which will run rabbitmq, nginx , and ruby. We have checked that the postal database has everything needed to run several servers with the same configuration provided that every cluster member would connect to the db server.
Our concern is naming and  ip addressing. Should every cluster member have the same name, eg: postal.domain.com ?? And obviously the ip addresses will be different, is there any problem with that? Our deploy will include a Cisco router configured as load balancer (in LSB mode).
Please advice us or let us know any suggestion to accomplish this project.
Regards
Diego

If all your traffic is going out via the router then in theory you can use its IP address as all your DNS records otherwise you would need to likely need to list multiple IP addresses in the DNS records which could be annoying to manage.
I think spam checkers will get annoyed if the server name doesn't match the domain name in one of the email headers so it will be important to get that right but I'm not sure if that is a Postal configuration thing or a server hostname thing.
There is nothing stopping you from running different parts of Postal on different members of the cluster if you have the numbers so that would also affect the DNS records.
Does that make sense?

Thanks for your response.  Yes, that will be the case, at the end, after passing out the load balancer/router, for the world the system will be seen as one ip address.  What we will try to deploy is a black box with an smtp server (several postal servers inside actually) exposed.
Regards,
Diego


A note on dealing with very old comments

The Reddit API doesn't return comments older than a few years. I'm not sure what the cutoff is, but I have 7+ year old comments that are ignored. So ereddicator or any similar program will run to completion without deleting the content. Adding to the confusion, new reddit will show the ancient comments while old.reddit.com will not.
This is not your bug, of course.
As a workaround: use the GDPR export feature and grab the comment id from comments.csv located in the export zip file, then delete them via the API. This worked for me:
import praw
import pandas as pd
from datetime import datetime, timedelta
import configparser
import os
import time
import pytz

def load_credentials(filepath='reddit_credentials.ini'):
    """
    Load Reddit API credentials from an INI file.
    """
    if not os.path.exists(filepath):
        raise FileNotFoundError(f"Credentials file not found: {filepath}")

    config = configparser.ConfigParser()
    config.read(filepath)

    required_keys = ['client_id', 'client_secret', 'username', 'password']

    if not config.has_section('reddit'):
        raise KeyError("Missing [reddit] section in credentials file")

    credentials = dict(config['reddit'])

    missing_keys = [key for key in required_keys if key not in credentials]
    if missing_keys:
        raise KeyError(f"Missing required credentials: {', '.join(missing_keys)}")

    return credentials

def load_comments(filepath):
    """
    Load comments from CSV file and convert date to datetime with UTC timezone.
    """
    df = pd.read_csv(filepath)
    df['date'] = pd.to_datetime(df['date'])
    return df

def delete_comments(reddit, comments_df, age_limit_years=7, dry_run=True, delay=2):
    """
    Delete comments older than specified age limit.

    Args:
        reddit: Initialized PRAW Reddit instance
        comments_df: DataFrame containing comments
        age_limit_years: Delete comments older than this many years
        dry_run: If True, only simulate deletion
        delay: Delay between API calls in seconds
    """
    # Create cutoff date with UTC timezone
    cutoff_date = datetime.now(pytz.UTC) - timedelta(days=age_limit_years * 365)
    old_comments = comments_df[comments_df['date'] < cutoff_date]

    print(f"\nFound {len(old_comments)} comments older than {age_limit_years} years")
    print(f"Cutoff date: {cutoff_date.strftime('%Y-%m-%d %H:%M:%S %Z')}")

    if dry_run:
        print("\nDRY RUN - No comments will be actually deleted")
        print("\nFirst 5 comments that would be deleted:")
        if len(old_comments) > 0:
            preview = old_comments.head()
            for _, row in preview.iterrows():
                print(f"- {row['date'].strftime('%Y-%m-%d')} - r/{row['subreddit']}: {row['body'][:100]}...")

    deleted_count = 0
    failed_count = 0
    skipped_count = 0

    for idx, row in old_comments.iterrows():
        comment_id = row['id']
        try:
            comment = reddit.comment(id=comment_id)

            # Verify the comment still exists and belongs to the authenticated user
            try:
                author = comment.author
                if author is None or author.name != reddit.user.me().name:
                    print(f"Skipping comment {comment_id} - Not found or not owned by you")
                    skipped_count += 1
                    continue

                if not dry_run:
                    comment.delete()
                    deleted_count += 1
                    print(f"Deleted comment {comment_id} from {row['date'].strftime('%Y-%m-%d')} in r/{row['subreddit']}")
                else:
                    deleted_count += 1
                    print(f"Would delete comment {comment_id} from {row['date'].strftime('%Y-%m-%d')} in r/{row['subreddit']}")

            except Exception as e:
                print(f"Error verifying comment {comment_id}: {str(e)}")
                failed_count += 1

        except Exception as e:
            print(f"Error accessing comment {comment_id}: {str(e)}")
            failed_count += 1

        # Sleep to avoid hitting rate limits
        time.sleep(delay)

    # Print summary
    print(f"\nSummary:")
    print(f"{'Would delete' if dry_run else 'Deleted'}: {deleted_count} comments")
    print(f"Failed: {failed_count} comments")
    print(f"Skipped: {skipped_count} comments")

    return deleted_count, failed_count, skipped_count

def main():
    try:
        # Load credentials
        credentials = load_credentials()

        # Initialize Reddit instance
        reddit = praw.Reddit(
            client_id=credentials['client_id'],
            client_secret=credentials['client_secret'],
            username=credentials['username'],
            password=credentials['password'],
            user_agent=f"script:comment_deleter:v1.0 (by /u/{credentials['username']})"
        )

        # Load comments
        print("Loading comments from comments.csv...")
        comments_df = load_comments('comments.csv')
        print(f"Loaded {len(comments_df)} comments")

        # Show date range of comments
        print(f"\nDate range of comments:")
        print(f"Earliest: {comments_df['date'].min().strftime('%Y-%m-%d')}")
        print(f"Latest: {comments_df['date'].max().strftime('%Y-%m-%d')}")

        # Ask for confirmation
        age_limit = 7  # years
        print(f"\nPreparing to process comments older than {age_limit} years...")

        # Dry run first
        print("\nPerforming dry run first...")
        delete_comments(reddit, comments_df, age_limit, dry_run=True)

        # Ask for confirmation before real deletion
        confirm = input("\nDo you want to proceed with actual deletion? (yes/no): ").lower()
        if confirm == 'yes':
            print("\nProceeding with deletion...")
            delete_comments(reddit, comments_df, age_limit, dry_run=False)
        else:
            print("\nOperation cancelled")

    except FileNotFoundError:
        print("\nError: Required file not found. Please ensure you have:")
        print("1. reddit_credentials.ini with your Reddit credentials")
        print("2. comments.csv with your comment data")
    except Exception as e:
        print(f"\nAn error occurred: {e}")

if __name__ == "__main__":
    main()```

Thank you for raising this. Currently trying to get my own exports so I can do some testing on my own accounts.

Added this feature now in v3.4: https://github.com/Jelly-Pudding/ereddicator/releases/tag/v3.4


[WIP] User Manual update -- secondary indexes

Description
Added a manual page to cover secondary indexes
Tests
Hmmm, not sure how that would work.
Links
TODO:  Jira
Checklist

[ ] Text Review
[ ] Squash Commits

People To Reivew
Puja
Authors of the indexes

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/702/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/704/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/714/Build result: FAILURE[...truncated 234.53 KB...][INFO] Apache Rya Spark Support ........................... SKIPPED[INFO] Apache Rya Web Projects ............................ SKIPPED[INFO] Apache Rya Web Implementation ...................... SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 06:59 min[INFO] Finished at: 2018-03-22T21:02:02+00:00[INFO] Final Memory: 203M/3162M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (check-licenses) on project rya.manual: Too many files with unapproved license: 1 See RAT report in: /home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.manual/target/rat.txt -> [Help 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn  -rf :rya.manualchannel stoppedSetting status of 00568c3f01e65ee507c6dd95bf6d6f9c5919a9f7 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/714/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing

Sorry folks, I goofed.  I fixed a tangle when merging and did a forced
push, changing one (some?) of the recent commits.  You may experience some
issues if you pulled/cloned from master early this week or last week.
For some reason after I merged a pull request, I had two copies of an old
commit and a third commit that had no changes in it.  I have seen similar
and fixed this before on my own repo.
I fixed it within five minutes of breaking it, so I thought it would only
affect people that pulled within that window.  But I might be wrong.  The
commit labeled:
"POM version fix. Closes #281"
has changed it's ID.  Also it appears after the commit I just merged.
Hopefully that is all that changed.
This should be only a minor annoyance for anyone that pulled/cloned from
master early this week or last week.  Let me know if I can help.
david.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/715/Build result: FAILURE[...truncated 234.53 KB...][INFO] Apache Rya Spark Support ........................... SKIPPED[INFO] Apache Rya Web Projects ............................ SKIPPED[INFO] Apache Rya Web Implementation ...................... SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 06:45 min[INFO] Finished at: 2018-03-26T20:36:10+00:00[INFO] Final Memory: 140M/2838M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (check-licenses) on project rya.manual: Too many files with unapproved license: 1 See RAT report in: /home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.manual/target/rat.txt -> [Help 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn  -rf :rya.manualchannel stoppedSetting status of d13f9b9474c87baf5127c7a5db4ff658c61e5090 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/715/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/788/Build result: FAILURE[...truncated 643.40 KB...][INFO] Apache Rya Spark Support ........................... SKIPPED[INFO] Apache Rya Web Projects ............................ SKIPPED[INFO] Apache Rya Web Implementation ...................... SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 14:49 min[INFO] Finished at: 2019-03-12T20:04:06+00:00[INFO] Final Memory: 204M/2990M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (check-licenses) on project rya.manual: Too many files with unapproved license: 1 See RAT report in: /home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.manual/target/rat.txt -> [Help 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn  -rf :rya.manualchannel stoppedSetting status of 62b74daa5e3d7028ea8c466dee0c39edd5373537 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/788/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/789/Build result: FAILURE[...truncated 630.90 KB...][INFO] Apache Rya Spark Support ........................... SKIPPED[INFO] Apache Rya Web Projects ............................ SKIPPED[INFO] Apache Rya Web Implementation ...................... SKIPPED[INFO] ------------------------------------------------------------------------[INFO] BUILD FAILURE[INFO] ------------------------------------------------------------------------[INFO] Total time: 12:58 min[INFO] Finished at: 2019-03-12T20:18:21+00:00[INFO] Final Memory: 158M/2556M[INFO] ------------------------------------------------------------------------[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (check-licenses) on project rya.manual: Too many files with unapproved license: 1 See RAT report in<EMAIL_ADDRESS>-> [Help 1][ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.[ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn  -rf :rya.manualchannel stoppedSetting status of a8fb79e0ffa9f486558eb8395d44605dff304cbd to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/789/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing


v4.2.0 not released on npm

Are you missed to publish version 4.2.0 on npm?
On npmjs.com is only version 4.1.3 available

Hi - we are preparing for the release right now! It will be live today or tomorrow.


[Pick 2.1](inverted index) fix memory leak in inverted index writer for array values

Proposed changes
Issue Number: close #xxx
Pick From #36144
Add ownership flag to Field's TokenStream value and Analyzer's Reader value, free them when tokenizer finalize.

run buildall


Fix error handler for maps and implemented converter of list fromatted configuration to record

Hello. I made a little patch for being able to resume encoding term to JSON in case of key is a bad string. Current implementation of error handler and jsx_parser:resume didn't work with maps. Also something was wrong with config, which was converted to list before calling error handler, but jsx_parser:resume expects record. I done this to use error handler for encoding binary (non printable) keys and/or values in maps and/or arrays. Examples in tests.

This PR is still compatible with the current code base. Any missing to get it merged and closed? @tsloughter


[BUG REPORT] New repo creation is unfriendly

Describe the bug
The message HANGAR RUNTIME WARNING: no repository exists at /some/path/__hangar, please use init_repo function makes me think my script is doing something wrong, even though the next thing that I do is call repo.init().
Additionally, if the path does not exist, there should be an option to have it be created.
A single default param exists=True flag could handle both cases, creating the directory and suppressing the init warning when set to False
Severity

Select an option:

[ ] Data Corruption / Loss of Any Kind
[x] Unexpected Behavior, Exceptions or Error Thrown
[ ] Performance Bottleneck

Thanks for the report @elistevens. there are still a few warnings/log messages which really need to be removed now that the project is approaching a more polished state. We'll add it to the todo list before 0.2

It's not particularly urgent, I don't think.

On another note Eli, what do you think about us naming the hangar folder with two underscores rather than a dot. /Foo/for/__hangar rather than foo/dir/.hangar
I'd like it to have the hidden one of the property and general sense of "don't go in this folder unless you know what you're doing" that a "."folder" provides, but considering that the amount of data contained in these folders can grow quite significant I didn't want to have somebody forgot about a random folder filling up their hard disk for no reason.
Also do you think we should put some sort read me in that top level folder explaining don't go in the hangar dir? Along with links to the project

I think that should be driven entirely by a clearly communicated expected use case. I was surprised that there was an extra directory level at all, but that's due to my expectations not being set properly.
Here's an example expected use case that I think could make sense (but is by no means the only one that would):

dir named .hangar/
lives next to .git/
adds a line to .gitignore to ignore .hangar (creates file if it doesn't exist)

That would look a bit odd on a global NFS mount, but it would still work. Alternatively, you could have some other expected use case, and you'd tailor it to match.
I think that I'd lean towards getting rid of the intermediate directory entirely, and refuse to init a repo in a non-empty directory. Then the user can put it wherever (/path/to/git/working/copy/.hangar/ if they like).  But that's based entirely on my weak understanding of what the intended use is.

So Im currently at the point where u consider the intermediate directory as definitely being important. The original idea was to model it like git. Keeping the code and data together makes a lot of sense intuitively. Sure, larger installations can do whatever they want, but to get started All you need to do is point the hangar client to you projects directory, and go. People understand that workflow.
It's also nice for CLI usage, since you don't have to navigate into the hangar repo, but instead operate at a level higher where you are less likely to be tempted to play around with files or inadvertently mess stuff up. I know we could solve this issue through static profiles (like kubectl) or some other mechanism, but the idea was to make a similar enough to get where people would just understand what to do.
The hangar repo folder structure is fairly sensitive to change. If you go around and start either moving files or writing data, things are going to really mess up.  At this point we would definitely notice that things are out of place / corrupted, (and it swill make you aware, though the stacktrace would be a mess) but there's not much recovery built in which accounts for user actions, only for the actions possible by the hangar core. My intention is to make it very clear that if you go inside and start making changes you do so at your own risk (which is why I like the  "." folders so much.)
For a NFS mounted repo, I guess it would look a bit odd, but it's not a use case I ever really considered. The server directory is named something different, so there's that atleast? But if you wanted to mount a real client repository on NFS I think you might just have to deal with it occurring the same way as git? It seems like an unnecessary bit of complexity to handle how someone wants their data container (essentially a database) named in their system.
I like the idea about modifying the .gitconfig file though!
Am I missing some big use case which would benefit from this?

It might be helpful to have a file written in the container director on initialization which basically say
This directory has been used to initialize a Hangar Repository

*Foo user* created this Repository on *bar-date* at *time* with and initial version *vX.Y.Z* of hangar.

Do not modify the contents of the ".hangar" folder under any circumstances. The contents are not meant to be understood by humans. Doing so will result in data loss / corruption.

More information can be found at *website*

This readme file can be safely modified or deleted at any time. Doing so will not affect the hangar Repository in any way.

.hangar.README.md IMO.

This has been addressed on #101! Thanks @elistevens!


Fix javadoc for hotrod default realm (these default to org.infinispan…

….client.hotrod.configuration.AuthenticationConfigurationBuilder#DEFAULT_REALM and org.infinispan.hotrod.configuration.AuthenticationConfigurationBuilder#DEFAULT_REALM)

Can you please rebase @rhusar ?

@tristantarrant Looks like someone force-pushed main... I recommend going to GH settings and prevent this entirely with a branch protection.

Merged, thanks


Wiki documentation fixes

I read through the entire wiki (except the API docs) today and figured that while I was at it I'd correct the various spelling and grammar errors and make a few minor changes to improve readability. A patch (against e8f6e2e) containing my fixes can be found at:
https://gist.github.com/brettle/10007085
You can download the raw patch at:
https://gist.githubusercontent.com/brettle/10007085/raw/6a4f14d57d34af504a2e0bc4e037a91974c03000/godot.wiki-readability.patch

Too bad this issue was overlooked for so long, now that the wiki is hosted on OpenProject with a new syntax it will be harder to apply this patch... I'll see what can be done (ping @StraToN and @TheoXD too).

Still interesting. I'll import this file locally and re modify it for the new Open Project wiki. And without forgetting giving credit to @brettle. Thanks to him

@her001 @StraToN If one of you has some time to check this patch and see what parts would still be relevant in the new documentation, that would be great! :)

Noted, added to my todo list. I wouldn't mind if someone else beat me to it.

Movable to the docs repo

Moved.


Add all project types to ddev import-files

Is there an existing issue for this?

[X] I have searched the existing issues

Is your feature request related to a problem?

https://github.com/ddev/ddev/issues/4190
https://github.com/ddev/ddev/pull/5005

Describe your solution
I noticed that not every project type supports ddev import-files:
https://github.com/ddev/ddev/blob/42969991212ff2c71fb91d7651d2ea17f8d1d749/pkg/ddevapp/apptypes.go#L75-L218
nodeps.AppTypeDjango4, nodeps.AppTypeLaravel and nodeps.AppTypePython are missing importFilesAction.
I guess it was just missed in the massive #5005.
Not sure about the specific actions for Django and Python, but Laravel should be the same as the generic PHP type.
Describe alternatives
No response
Additional context
No response

Not all project types have an explicit location for user-generated files. For example, with Craft CMS and Laravel it's completely a developer choice. Same with php.
However, if upload_dirs is specified, ddev import-files should still work.

However, if upload_dirs is specified, ddev import-files should still work.

Unfortunately, it doesn't work:
# prepare a test folder with a test file
mkdir test && cd test && mkdir test_folder && touch test_folder/file.txt

# try to import files for PHP project type
ddev config --auto --upload-dirs=test_folder2 --project-type=php
ddev import-files --source=test_folder
Successfully imported files for test

# try to import files for Laravel project type
ddev config --auto --upload-dirs=test_folder2 --project-type=laravel
ddev import-files --source=test_folder
Error: failed to import files for test: this project type (laravel) does not support import-files

nodeps.AppTypeDjango4, nodeps.AppTypeLaravel and nodeps.AppTypePython are missing importFilesAction.

I intended to fix this with a separate PR but looks like I'll need it to get tests to pass for the panic PR, https://github.com/ddev/ddev/issues/5126