Milliseconds are handled incorrectly when passing dates from JS to Rust

This program
use quick_js::Context;

fn main() {
    let context = Context::new().unwrap();
    let result = context
        .eval("new Date('2019-08-12T21:10:12.123Z')")
        .unwrap();
    println!("result: {:?}", result);
}

outputs
result: Date(2019-08-12T21:10:12.000123Z)

It might be related to this (fixed) bug: https://www.freelists.org/post/quickjs-devel/Bug-report-incorrect-handling-of-fractions-of-seconds-in-dates-parse.

The problem seems to be not related to QuickJS date parser:
use chrono::{offset::TimeZone, DateTime, Utc};
use quick_js::{Context, JsValue};

fn main() {
    let date = "2019-08-12T21:10:12.123Z";
    let timestamp = DateTime::parse_from_rfc3339(date)
        .map(|timestamp| Utc.from_utc_datetime(&timestamp.naive_utc()))
        .map_err(|err| format!("Unable to deserialize date: {}", err))
        .unwrap();
    println!("original: {:?}", date);

    let context = Context::new().unwrap();
    context.eval("identity = x => x").unwrap();
    let result = context
        .call_function("identity", vec![JsValue::Date(timestamp)])
        .unwrap();
    println!("convered: {:?}", result);
}

prints
original: "2019-08-12T21:10:12.123Z"
convered: Date(2019-08-12T21:10:12.000123Z)

It might be related to this (fixed) bug

Nah I was just tired and hasty yesterday.
Will release 0.2.2 with the fix right after it's merged.


Repository-specific string match

Users should be able to preface their search strings with the name of a known component type or provider (e.g., github:,  npm:, npmjs:, …) and have the search results restricted to just those types or providers.

This is not yet in place but probably the new implementations of definitions query builder on the service could help to fullfill this requirement.

@dabutvin could you confirm that using some of the params exposed from the definition API, we are able to make query like the one specified?
I guess it could be enough to pass a provider= param together with the string

good point @storrisi
@fossygirl do you still want to light up this kind of syntax in the current state of things? Is this specifically for https://clearlydefined.io/definitions ?

@fossygirl any updates on this?

@jeffmcaffer probably has thoughts here.

For the future reference, if we search "Query Parser Syntax github" on Google. There are plenty of JS repo we could use to parse the text field and get for example this parsed provider:npm jquery license:none
For the first implementation we agreed on only providing a search by provider, therefore the above "Query Parser Syntax" might be overkill for a single search parameter

@fossygirl may I ask you an update on this? Do we still need this implementation considering the current state of things?


Add resources section; add PDFs

Add links to 2022 Firefwise certificate, 3-year plan, and community assessment.

Thanks @abs! We already have a resources page, so should we name this heading differently to avoid confusion, or perhaps add the resources to the existing page?
It may be easier on our git repo's size if we hosted binary files on the firewise gdrive, but these are relatively small so no big deal.

@stefanv Ah, sorry, didn't notice. Renamed to 'Certificate and program documents'. Roger on PDFs - lets store future files and versions elsewhere.

Excellent, thanks Andrei!


⚠ Bofrost.at has degraded performance

In 41385ed, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1165 ms

Resolved: Bofrost.at performance has improved in f195857.


⚠ Bofrost.at has degraded performance

In 7dc063c, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1137 ms

Resolved: Bofrost.at performance has improved in bd0e14e after 6 minutes.


There appears to be a memory leak during the training phase？

I train model  from scratch  on the V100 , the cpu memory grew during the training process, and after about 10,000 more steps, the cpu memory of the machine was used up, 512GB of memory
How do I locate this problem, thank you
Using the following script：
python scripts/train.py --config scripts/configs/octo_pretrain_config.py:vit_s

The TFDS data loader dynamically allocates memory during training and we have observed similar "memory grows over time" behavior. Here are a few knobs you can turn to control memory usage:

reduce the amount of parallelism during data loading by reducing traj_transform_threads, traj_read_threads and num_parallel_calls inside frame_transform_kwargs (see examples/05_dataloading.ipynb for some explanations) -- this will reduce memory usage but potentially also reduce training speed
remove eval during training -- running eval uses memory and for some of our larger runs we remove it during training to avoid memory overflow and then run it post-hoc to visualize the model's performance
reduce the size of the shuffle buffer -- this is the most direct way to cut down memory usage, but careful: too small of a shuffle buffer will negatively impact training dynamics and we found models trained with a shuffle buffer of only 100k samples to perform worse

All that being said, we train our vit_s models on a TPUv4-8 machine with 400GB of memory, so if your machine has 512GB it should be able to fit it, maybe start by disabling evaluation during training.

Hello, I have encountered a similar issue. I am training on four A100 GPUs using PyTorch's DataLoader (https://github.com/octo-models/octo/blob/main/examples/06_pytorch_oxe_dataloader.py) with a shuffle buffer size of 500K. I found that each process is using over 200GB of memory, and the total memory usage of the four processes exceeds 1TB, leading to out of memory errors. Do you have any experience with this? How can I resolve this issue?
Thanks in advance!

It's important that you keep the number of workers to 0, so that all the parallelism is handled by TFDS. Basically you only want a single data loading process (and shuffle buffer) per compute node, ie in your case only one. I'd expect total memory usage for a single loader with 500k shuffle buffer to not exceed 300GB.

Thank you for your response!
I have one more question. I am using PyTorch's DistributedDataParallel for multi-GPU training. How can I do data loading in only one process? I noticed that in the examples of DDP, each process has its own dataloader.


Use short major version tags

It's common to reference GitHub Actions only with major version number, not full semver version. This way, end users keep getting the latest minor/patch release without any change.
For example, actions/checkout action, you have these v2/v3 tags, so you use it like actions/checkout@v3, not<EMAIL_ADDRESS>
P.S. Alternative is to maintain v1 branch with same content as latest v1.x.y release.

Hi @gonadarian ,
thank you for the report and tip we will update our scripts and should have this reflected with the next version update.
best regards,
Steffen


Documentation/Examples for streaming requests and responses

Describe the problem
As of #5291 SvelteKit now supports using ReadableStream to read the request body in endpoints and supports returning it as a response body, however there are currently no examples in the docs on how to do common things, for example reading files submitted from a multipart/form-data as a stream and write them to disk, reading files from disk and returning them as a stream, etc.
Describe the proposed solution
Adding examples for common use-cases of streaming in the docs
Alternatives considered
No response
Importance
nice to have
Additional Information
I would help with this myself, but I honestly couldn't figure it out, and I wouldn't trust the code I write to be up to the standard of the docs. I specifically wanted to handle super-large (over 1gb) file uploads that would be in a multipart/form-data request with other fields but couldn't figure it out.
Also feel like I should mention this: in #3419, Rich mentions having utility functions for handling multipart/octet streams, so if the maintainers are planning on adding them then it would make sense to not add documentation until they are added, as they would simplify any code that could be written without them.

Don't have time to write the docs myself, at least right now — but here are a few things to know about readable streams and multipart/form-data:

The ReadableStream interface doesn't provide a read() method directly; instead, you call getReader() to get a Reader interface that has a read() method.
There are two different reader modes, "default" mode (which creates a Buffer object for you) and "bring-your-own-buffer" or "BYOB" mode, in which you're expected to pass a Buffer object to the reader's read() method. The default mode is easier to handle, but the BYOB mode is better for saving memory in some situations. If you already have a Buffer object that you want to put the data into, use BYOB mode, otherwise just use default mode.
Each time you call read(), you get a "chunk". Chunks are not guaranteed to end at a convenient place. They could end right in the middle of a UTF-8 byte sequence, e.g. the last byte of one chunk could be 0xE2, the next chunk could be a single byte containing 0x80, and the last chunk could start with 0xA2. If you assemble those properly, you'll be able to reassemble the 0xE2 0x80 0xA2 sequence, which is UTF-8 for the • character. Do it wrong and you could end up with two or three � characters instead.
When processing multipart/form-data content, you're looking for the sequence CRLF, two hyphens, the boundary marker, optionally two more hyphens to signal "end of multipart data", and another CRLF. So you're looking for either 0x13 0x10 0x2D 0x2D(boundary)0x13 0x10, or else 0x13 0x10 0x2D 0x2D (boundary) 0x2D 0x2D 0x13 0x10` (assuming your incoming data is in UTF-8). Remember that chunks can be split anywhere, so don't assume that the boundary sequence will be found in a single chunk. It could be split across multiple chunks, even.

Whoever produces sample code for this will need to make sure the sample code doesn't fall into the most common pitfall of stream-handling, which is assuming that chunk boundaries will be nice. A chunk boundary that splits a UTF-8 sequence or a multipart/form-data boundary marker is going to cause issues to naive code, and the examples should probably show how to do it right.

Having said that, here's a general guideline to handling large file uploads in a form with other data:

Create a stream reader.
Start searching chunks for the boundary sequence (obtained from the headers)
Every time you encounter a CRLF during the search, and you have not found a boundary sequence yet, you can safely consider the data up to that CRLF to be part of one section, rather than a potential boundary, and handle it appropriately (write it to the file, save it as a form value, whatever the previous boundary header said to do).
When you encounter a CRLF followed immediately by a couple of hyphens, don't just blindly compare the next bytes to the boundary sequence. They might be truncated. Instead, start comparing those bytes one-by-one to the boundary sequence, with one of three outcomes:

They match a boundary and are complete. The previous section is complete; you can close the WritableStream to the file, or save the form data, or whatever.
You find a character that doesn't belong to the boundary sequence. Everything from the CRLF to that character is therefore part of the previous section, which is not yet complete. Stop matching against the boundary sequence, add what you've found so far to the previous section, and go back to scanning for CRLF characters.
You run out of characters before completing the boundary sequence. Read the next chunk and continue this step, until either the boundary sequence is a complete match (so the previous section is complete, as in step i) or the boundary sequence fails (in which case this data is part of the previous section, as in step ii).

After a boundary marker is found, you're at the start of a new MIME section, so expect some headers followed by a blank lines. Those headers will tell you what kind of thing is coming up and how to handle it. E.g., if it's a file, create a StreamWriter to write the contents somewhere, and so on.

I'd love to use an existing library for this rather than rolling our own multipart form data parser, even if we abstract it somehow. Unfortunately the ones that I know of (busboy, formidable) are wedded to Node AFAICT.
Anyone know of something more modern, that can take a Response object (or a Headers and a ReadableStream, I guess) and do something useful with it?

Streaming A Large Dataset With A Pipe To Response Writable would be handy. So documentation with an example of a large file being read by an endpoint, converted to a readable stream, and then being fetched by a page is a common use case, avoiding filling up memory. I tried some usual code that would work in Express or Hyper-Express with piping a readableStream, but I get a dest.on is not a function type error.
TypeError: dest.on is not a function
at Stream.pipe (node:internal/streams/legacy:33:8)

Something like how Express or Hyper-express gives examples for piping would be great.
https://github.com/kartikk221/hyper-express/blob/master/docs/Examples.md
Makes it easy to pipe to a stream to another stream etc. etc.
readStream.pipe(transformStream).pipe(response).on('finish', () => {
                    console.log(`Finished transforming the contents of ${filePath} and piping the output to the response.`);
                });

This seems like a common use case.

I hope this is relevant here.
I am importing/using an npm package on a server endpoint and getting the errors
response.body.getReader is not a function.
From the conversations I see that this might be related here - how would I with the new feature enable the package to use ReadableStream?

I've been testing Server Sent Events (SSEs) with Kit recently. It's almost working properly, but the non-streamable Response objects obviously prevent the connections from staying open.
In a comment on a related issue, Rich said SSEs are a special case of ReadableStreams. Now that ReadableStreams appears to be implemented, how can we use this to make SSE connections stay open?
Ideally, we would be able to have access to all the res and req methods so that we can properly detect and respond to stream states, too, but simply keeping the connection open would be a great leap forward.
Thank you @Rich-Harris and the rest of the team for all the work you've done on this project.

@JuliaBonita
For SSE it is enough to return ReadableStream as response body. I can give a more complex example based on TransformStream, which subscribes to EventEmitter and unsubscribes when connection is closed.
+server.js
import { createSSE } from './sse';
import { bus } from './bus';

/** @type {import('./$types').RequestHandler} */
export async function GET({ request }) {
	// does not have to be a number, this is just an example
	const last_event_id = Number(request.headers.get('last-event-id')) || 0;

	const { readable, subscribe } = createSSE(last_event_id);

	subscribe(bus, 'time');
	subscribe(bus, 'date');

	return new Response(readable, {
		headers: {
			'cache-control': 'no-cache',
			'content-type': 'text/event-stream',
		}
	});
}

sse.js
export function createSSE(last_id = 0, retry = 0) {
	let id = last_id;
	const { readable, writable } = new TransformStream({
		start(controller) {
			controller.enqueue(': hello\n\n');

			if (retry > 0) controller.enqueue(`retry: ${retry}\n\n`);
		},
		transform({ event, data }, controller) {
			let msg = `id: ${++id}\n`;
			if (event) msg += `event: ${event}\n`;
			if (typeof data === 'string') {
				msg += 'data: ' + data.trim().replace(/\n+/gm, '\ndata: ') + '\n';
			} else {
				msg += `data: ${JSON.stringify(data)}\n`;
			}
			controller.enqueue(msg + '\n');
		}
	});

	const writer = writable.getWriter();

	return {
		readable,
		/**
		 * @param {import('node:events').EventEmitter} eventEmitter
		 * @param {string} event
		 */
		async subscribe(eventEmitter, event) {
			function listener(/** @type {any} */ data) {
				writer.write({ event, data });
			}

			eventEmitter.on(event, listener);
			await writer.closed.catch(() => { });
			eventEmitter.off(event, listener);
		}
	};
}

bus.js
import { EventEmitter } from 'node:events';

export const bus = new EventEmitter();

setInterval(() => {
	bus.emit('time', new Date().toLocaleTimeString());
}, 2e3);

setInterval(() => {
	bus.emit('date', new Date().toLocaleDateString());
}, 5e3);

@repsac-by Thank you very much for your help. I appreciate this and will spend more time learning the Streams API if necessary, but it's a totally different API and much more complex than the simple SSE API. The simplicity of the SSE API for simple server-push applications is the most significant feature that differentiates it from WebSockets. If we have to use the more complicated Streams API in Kit, then it seems we might as well use WebSockets and assume our apps will use bi-directional channels even if we only need data pushed from the server to the client.
My assumption (maybe incorrect) was that Kit would include a way to use the actual SSE API with the same simple functionality that we have with vanilla Node. Your example is excellent for the Streams API, but it still requires a significant learning curve to achieve the equivalent functionality and reliability that is trivially easy to achieve with SSEs, which is sufficient for the vast majority of text-based server-push applications.
So before I go down the more complicated Streams API path, can you please confirm that there is no efficient way to implement SSE in Kit to keep connections open and manage the connections like we can easily do in pure Node?

I just looked at the source code for the Streams API "Hello World" example. It requires 80 lines of code (including markup) just to display "Hello World" on the page. Swimming through all the Streams API docs is a deep rabbit hole. Clearly the Streams API is not ergonomic, which is why very few people actually understand or use it. It's also overkill for virtually all SSE text-based applications.
I bet over 95% of the Svelte community will never take the time to learn the Streams API well enough to use it effectively, which makes Kit much less likely to be used in large-scale apps. So I hope there's a way to use actual SSEs in Kit (including proper event handling on the server like with Express).

@JuliaBonita
So, here is the ReadableStream for SSE in a few lines.
export async function GET() {
	const readable = new ReadableStream({
		start(ctr) {
			this.interval = setInterval(() => ctr.enqueue('data: ping\n\n'), 3000);
		},
		cancel() {
			clearInterval(this.interval);
		}
	});

	return new Response(readable, {
		headers: {
			'content-type': 'text/event-stream',
		}
	});
}

@repsac-by Thank you. The second example helps to show that the Streams API is basically a transport mechanism to stream any data flow, including higher level protocols like SSEs. This should be enough for me to figure out how to apply it to my app.
In the future, I suspect Svelte devs will ask for a more simplified interface for streams and SSEs like Express provides, but for now, at least I can see a path forward. Thank you.

Okay so after a bit of playing around this is what i came up with:
Send streams:
// src/routes/stream/+server.ts
import { stream } from '$lib/stream';

export const GET: RequestHandler = async () => {
	const fsread = createReadStream('image-endpoint.jpeg');

	return stream(fsread, {
		status: 200,
		headers: {
			Cache: 'no-cache',
			'Content-Type': 'image/jpeg'
			// 'Content-Disposition': `attachment; filename=${'testfile'}.jpeg`
		}
	});
};

Receive streams:
// src/routes/stream/+server.ts
import { getStream } from '$lib/stream';

export const POST: RequestHandler = async ({ request }) => {
	const stream = getStream(request);
	if (!stream) {
		throw error(400);
	}

	const fswrite = createWriteStream('image-endpoint.jpeg', 'utf-8');

	stream.pipe(fswrite);

	return new Response();
};

Handle form files as stream:
// src/routes/stream/+server.ts
import { getStream } from '$lib/stream';

export const actions: Actions = {
	async default({ request }) {
		const formdata = await request.formData();
		const file = formdata.get('file') as Blob;

		const stream = getStream(file);
		if (!stream) {
			throw error(400);
		}

		const fswrite = createWriteStream('imagefile-form.jpeg', 'utf-8');

		stream.pipe(fswrite);

		return {};
	}
};

The actual implementation:
import { Readable } from 'node:stream';

export function stream(stream: Readable, init?: ResponseInit | undefined) {
	const body = new ResponseStream(stream);
	return new Response(body, init);
}

export function getStream(requestOrBlob: Request | Blob) {
	if (requestOrBlob instanceof Request) {
		return RequestStream.fromRequest(requestOrBlob);
	} else {
		return RequestStream.fromBlob(requestOrBlob);
	}
}

class RequestStream extends Readable {
	private readonly _reader: ReadableStreamDefaultReader;

	constructor(readable: ReadableStream) {
		super();
		this._reader = readable.getReader();
	}

	async _read(): Promise<void> {
		const { done, value } = await this._reader.read();
		if (done) {
			this.push(null);
		} else {
			this.push(value);
		}
	}

	static fromRequest(request: Request) {
		const stream = request.body;
		if (stream instanceof ReadableStream) {
			return new RequestStream(stream);
		}
		return false;
	}

	static fromBlob(blob: Blob) {
		if (typeof blob?.stream !== 'function') return false;
		const stream = blob.stream();
		if (stream instanceof ReadableStream) {
			return new RequestStream(stream);
		}
		return false;
	}
}

class ResponseStream extends ReadableStream {
	constructor(stream: Readable) {
		super({
			async start(controller) {
				for await (const chunk of stream) {
					controller.enqueue(chunk);
				}
				controller.close();
			}
		});
	}
}

I´m not sure if node:stream is available with all platforms/adapters.

@repsac-by Your "SSE in a few lines" comment helped me to push ticker data from server to client.
More here: https://stackoverflow.com/questions/74330190/how-to-respond-with-a-stream-in-a-sveltekit-server-load-function/74336207#74336207

@voscausa you forgot something
/** @type {import('./$types').RequestHandler} */
export function GET({ request }) {
+  const ac = new AbortController();

  console.log("GET api: yahoo-finance-ticker")
  const stream = new ReadableStream({
    start(controller) {
      tickerListener.on("ticker", (ticker) => {
        console.log(ticker.price);
        controller.enqueue(String(ticker.price));
-      })
+      }, { signal: ac.signal });
    },
+      cancel() {
+          ac.abort();
+      },
  })

  return new Response(stream, {
    headers: {
      'content-type': 'text/event-stream',
    }
  });
}

OK. I expected to use controller.abort() in the +page.svelte ??
 onDestroy(() => {
    controller.abort();
  });

@voscausa you can read the MDN docs here on the AbortController interface to understand what the signal does.

Just found out about unstable_parsemultipartformdata in the remix docs.
Implementation can be found here https://github.com/remix-run/remix/blob/main/packages/remix-server-runtime/formData.ts
Pretty sure that can be used in sveltekit aswell.

unstable_parsemultipartfoemdata looks like what was needed. I'll fiddle around and see if I can come up with something that works

Would be really great to have a small consensual example here while waiting for an official doc!

Here is how I get SSE working for me
routes/sse/+server.ts
function delay(ms: number): Promise<void> {
  return new Promise((res) => setTimeout(res, ms));
}

export function GET() {
  const encoder = new TextEncoder();
  const readable = new ReadableStream({
    async start(controller) {
      for (let i = 0; i < 20; i++) {
        controller.enqueue(encoder.encode('hello'));
        await delay(1000)
      }
      controller.close()
    }
  });

  return new Response(readable, {
    headers: {
      'content-type': 'text/event-stream',
    }
  });
}

routes/sse/+page.svelte

{#each result as str}

{str}

{/each}

any updates on this?

Example using EventSource in the client.

Supported by Chrome 6+, Safari 5+
Provides automatic re-connection, etc.

  {#each messages as message (message)}

{message}

  {/each}

When the server responds with a custom event 'close', the client closes the EventSource and does not reconnect.
// src/routes/sse/+server.ts

const duration = 5 * 1000; // 5 seconds

export const GET = () => {
  const encoder = new TextEncoder();
  const readable = new ReadableStream({
    start: async (controller) => {
      const startedAt = new Date();
      while (true) {
        const message = new Date().toISOString();
        controller.enqueue(encoder.encode(`data: ${message}\n\n`));
        const elapsedFor = Date.now() - startedAt.valueOf();
        if (elapsedFor > duration) break;
        await new Promise((resolve) => setTimeout(resolve, 1000));
      }
      controller.enqueue(encoder.encode('event: close\ndata:\n\n'));
      controller.close();
    }
  });
  return new Response(readable, { headers: { 'content-type': 'text/event-stream' } });
};

The server does not check for client disconnection in this example. Reference the following issue: https://github.com/sveltejs/kit/issues/11751

I need a way to detect when a client has disconnected, otherwise I'll be wasting I/O time on emitting data to a stream that nobody's ever going to read.

Since the OP asked for a multipart/form-data example:

Upload image file to a Node.js (dev) server
Pipe readable stream into a writable stream
Set filename extension based on the MIME type.

<form method="post" use:enhance enctype="multipart/form-data">
  <input type="file" name="image" accept="avif,jpg,jpeg,png,webp" required />
  <button>Submit</button>
</form>

import { error } from '@sveltejs/kit';
import { randomUUID } from 'node:crypto';
import { createWriteStream } from 'node:fs';
import { Readable } from 'node:stream';
import type { ReadableStream } from 'node:stream/web';
import { file, parse, picklist } from 'valibot'; // Optional

export const actions = {
  default: async ({ request }) => {
    if (!request.headers.get('content-type')?.startsWith('multipart/form-data')) error(415);

    const formData = await request.formData();

    // File
    const image = parse(file(), formData.get('image'));

    // "image/avif" | "image/jpeg" | "image/png" | "image/webp"
    const type = parse(
      picklist(['image/avif', 'image/jpeg', 'image/png', 'image/png', 'image/webp']),
      image.type
    );

    // TODO Throw error if `image.size` is too large;

    const filename = `${randomUUID()}.${type.substring(6)}`;

    // Same name, but completely different types:
    // Argument of type 'ReadableStream<Uint8Array>'
    // is not assignable to parameter of type 'ReadableStream<any>'.
    const readable = Readable.fromWeb(
      // globalThis.ReadableStream<Uint8Array>
      // node:stream/web.ReadableStream
      image.stream() as ReadableStream
    );
    const writeStream = createWriteStream(filename);
    readable.pipe(writeStream);

    try {
      await new Promise((resolve, reject) => {
        readable.on('end', resolve);
        readable.on('error', reject);
      });
    } catch {
      // TODO Handle upload error
    } finally {
      readable.removeAllListeners();
      readable.destroy();
      writeStream.end();
    }
  }
};

[!NOTE]

Readable.fromWeb is supported from Node.js 17+
No external library is used. Valibot is purely optional.

[!IMPORTANT]

This probably works in Node.js environment only.
But then, how would you save files locally in a edge function?
Maybe file based example should be different per platform.

Would this work? @Rich-Harris


Command line argument: set hostname

Add a command line argument to set the hostname being broadcast

Implemented in #8


Cancel button on History tab seems unnecessary

(2.8dev) On the History tab, if I click on an execution, I see this:

Note the "Rerun" and "Cancel" buttons. There does not seem to be any situation where I can use that Cancel button. The only time it makes sense is if I click Rerun, and then click Cancel before execution. But clicking Rerun brings up a new box.
This Cancel button might make sense if I could edit parameters without having a separate popup.

When on a long-running execution, such as a workflow, the cancel button will be available.

Good point. Maybe a tooltip makes sense? Will close this then

Might make sense to only show it when it is available, rather than disabling it? Thats probably a question for @adrifolio though.

I think it makes sense to show it only when is usable. Something else that would be beneficial is to show a loading icon on the rerun button while the workflow runs. For quick workflows it wouldn't make much of a difference, but for long workflows it would make the interaction more clear.


Correção de texto

Por favor, corrijam a concordância nominal ..
Proteger fisicamente a infraestrutura de IoT: os pior ataques de segurança contra .......

Detalhes do documento
⚠ Não edite esta seção. É necessário para a vinculação do problema do docs.microsoft.com ➟ GitHub.

ID: f75d4bfd-d824-01fa-bf9b-c8597224ed8d
Version Independent ID: 90167863-8849-9a69-dfb0-fd693b34d2b3
Content: Práticas recomendadas de segurança de Internet das Coisas | Microsoft Docs
Content Source: articles/security/azure-security-iot-best-practices.md
Service: security
GitHub Login: @TomShinder
Microsoft Alias: yurid

@yararl0306 Thank you for your feedback! We have assigned this issue to the author to investigate further and take the right course of action.

@amanarneja, can you please assign this issue to @DavidPrio

Hello, @yararl0306,
Your feedback is much appreciated.
Since your comments are specific to the localization of the documentation, we have moved this issue to the corresponding localization repository and we will close this one.
Please review the status of this issue here https://github.com/MicrosoftDocs/azure-docs.pt-br/issues/93.
Kind regards,
Microsoft DOCS International Team


Separate too much business logic from login page

Is your feature request related to a problem? Please describe.
Is it possible to move extra business logic from login page file to dedicated file?
Describe the solution you'd like
Service file to handle code for login page.
Describe alternatives you've considered
Leave code as it was.
Additional context

Not needed on this stage of project.


Failures during installer creation aren't failing the build

https://mssqltools.visualstudio.com/CrossPlatBuildScripts/_build/results?buildId=42865
This build had a failure in the create system installer task but that didn't fail the build. We need to make sure the build fails if any errors such as that occur.
Reading file (WizardImageFile)
   File: e:\mssqltoolsagentworkspace\87\s\resources\win32\inno-big-100-insiders-insiders.bmp
Error on line 23 in E:\mssqltoolsagentworkspace\87\s\build\win32\code.iss: Could not read "e:\mssqltoolsagentworkspace\87\s\resources\win32\inno-big-100-insiders-insiders.bmp".

Error: The system cannot find the file specified.
Compile aborted.

updated the build pipeline to enable "Fail on Standard Error"


How about the speed? Only 10fps on a Titan Xp

Hello, I run the project with only 10 fps on a titan xp, which is much slower than mentioned in README.md. Does anyone know how to accelerate? How to achieve a speed like real-time tracking?

Hello, I have the same question too !!

comment these in demo.py:
cv2.imshow('SiamMask', im)
key = cv2.waitKey(1)

@drahmad89  have you tested this model with VOT2016 dataset using this command (python ../../tools/eval.py --dataset VOT2016 --tracker_prefix C --result_dir ./test/VOT2016) ? Please let me know, I am trying to do so but facing the following error.  Did you download any tracker file to produce results?

@SamihaSara ,When I finshed the vot2018's test and want to run the statement"python ../../tools/eval.py --dataset VOT2018 --tracker_prefix C --result_dir ./test/VOT2018",I met the same question with you .Have you solved the problem?

I am also getting slow results - max 8 FPS on GPU. Any advice?

@liuwenbo4795 , hello, have you fixed it ?

@haishibei I have not figured out a solution yet

@GOBish  ok, thanks. I test it at VOT2016,the speed is  about 20fps-30fps, but I failed test it at VOT2018, and encountered the same question"divided by zero", have  you fixed it? @SamihaSara @liuwenbo4795 , waiting for your  reply, thanks


Full bootstrap example?

Great clean implementation, it would be nice if you guys could add a full bootstrap example.
Like on the demo page, hacking it together from the source of your own demo but a clean implementation provided in the package would be a welcome addition.

Do you mean something that uses the bootstrap framework or something else? This project is meant to be a UI-less editor that you can integrate into other toolkits to match your UI. If someone would like to do a Bootstrap integration that would be great! But it belongs in a separate project, much like this Angular integration.


Error message for dummy procedure argument.

In the following module example, declaring implicit none to detect implicit types for variables like "x" generates the following error message for the dummy procedure f: "No matching declaration found for argument "f"". Also, if the type declaration of "x" is uncommented then the warning message "Variable "x" masks variable in parent scope" appears, for variable "x" in the interface.
module test

    implicit none

    contains

        subroutine foo(f)

            interface
                function f(x)
                    real, intent(in) :: x
                    real :: f
                end function
            end interface
            !real :: x

            x = 1.
            print*, f(x)

        end subroutine

end module

I am also seeing this issue in my fortran subroutines that take interfaces as dummy arguments.

I have the same issue.

I have the exact same issue. For 1D real functions, using real(dp), external :: f instead of an interface block does not give me the error, but I'd prefer not to rely on an outdated (F77) method for passing function names as arguments.
Note: all of my source files compile and run perfectly fine with the interface block method of declaring the dummy procedure argument.

Okay so I finally got some spare time and have a fix for this. I will open a PR but given that the repo hasn't been updated in more than a year I will most likely be authoring a fortls2 or a dev version of fortls so people can start using a more up to date version.


make watching alpha CRDs optional

The ownership of CRD installation is not tied to a single entity https://gateway-api.sigs.k8s.io/guides/crd-management/#who-should-manage-crds This results in multiple entities taking ownership of CRD installation

infra users
implementations
cloud providers

This complicates things for implementations who may not know which version and release of CRDs are installed, so this PR makes watching alpha versioned CRDs optional

Even Envoy Gateway specific CRDs have been made optional to solve the use case where users want to only configure Gateway API resources

GRPCRoute is the only exception, which is v1, but has been made optional because it just graduated to v1 in v1.2 but a lot of cloud providers or service mesh implementations have not moved to v1.2

Fixes: https://github.com/envoyproxy/gateway/issues/3387

manually tested this by

Installing v1.0.0 Gateway API CRDs from the Stable Channel

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml

Installed Envoy Gateway using Helm and the --skip-crds option

helm install eg oci://docker.io/arkodg/gateway-helm --version v0.0.0-test3 -n envoy-gateway-system --create-namespace --skip-crds

Installed the QuickStart

kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v1.1.2/quickstart.yaml -n default

Tested it

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/get
*   Trying <IP_ADDRESS>:80...
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 80
> GET /get HTTP/1.1
> Host: www.example.com
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Sat, 26 Oct 2024 00:38:16 GMT
< content-length: 456
<
{
 "path": "/get",
 "host": "www.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/8.4.0"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "<IP_ADDRESS>"
  ],
  "X-Forwarded-Proto": [
   "http"
  ],
  "X-Request-Id": [
   "8f3833ab-c84c-4bb9-8f23-a3b165715ba0"
  ]
 },
 "namespace": "default",
 "ingress": "",
 "service": "",
 "pod": "backend-69fcff487f-d4fpj"
* Connection #0 to host <IP_ADDRESS> left intact


Updated DeveloperGuide.adoc file

Added user stories and use cases for the 'Notes Management' feature in DeveloperGuide.adoc file.

Pull Request Test Coverage Report for Build 33

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 92.811%

Totals

Change from base Build 32:
0.0%

Covered Lines:
1433

Relevant Lines:
1544

💛  - Coveralls


Download via appveyor of binaries is not working

Hello,
https://ci.appveyor.com/project/Rem0o/fancontrol-adlx
Build started git clone -q --branch=main https://github.com/Rem0o/FanControl.ADLX.git C:\projects\fancontrol-adlx git checkout -qf ee0ff5c0b7e4dc7a5[2](https://ci.appveyor.com/project/Rem0o/fancontrol-adlx#L2)ff244669a7e0fc0d4ae20c dotnet restore   Determining projects to restore...   Restored C:\projects\fancontrol-adlx\FanControl.ADLX.csproj (in 165 ms). msbuild "C:\projects\fancontrol-adlx\FanControl.ADLX.sln" /verbosity:minimal /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" Microsoft (R) Build Engine version 16.11.2+f[3](https://ci.appveyor.com/project/Rem0o/fancontrol-adlx#L3)22596[4](https://ci.appveyor.com/project/Rem0o/fancontrol-adlx#L4)2 for .NET Framework Copyright (C) Microsoft Corporation. All rights reserved.   FanControl.ADLX -> C:\projects\fancontrol-adlx\bin\Release\net48\FanControl.ADLX.dll Discovering tests...OK Collecting artifacts... Found artifact 'bin\Release' matching 'Bin\Release' path Uploading artifacts... [1/1] bin\Release.zip ([7](https://ci.appveyor.com/project/Rem0o/fancontrol-adlx#L7)7,4[8](https://ci.appveyor.com/project/Rem0o/fancontrol-adlx#L8)7 bytes)...[10](https://ci.appveyor.com/project/Rem0o/fancontrol-adlx#L10)0% Build success
All I can see is a console but no download buttons nor any automatic download.
What supposed to happen over there ?
Thanks!

Nothing you can do, the git repository owner need to rebuild the release since all artifacts are deleted after one month.
Can you publish last release in proper section please ? @Rem0o

@midoriiro Thank you for the information ! I guess I will wait for @Rem0o to find a solution.

I would love to give this a try! Please trigger a build @Rem0o ♥

@dayblox  done

@Rem0o thanks for triggering the build. Any reason why this release is not automated, or at least release artifact is not provided easily like your other repository?
I could give it a look, even if this plug-in is an experimental one. It could be useful to many people to experiment with their AMD GPU and FanControl. What do you think?

@midoriiro
Because until this project goes forward, I do not feel comfortable having regular users use this.  CPU consumption is way too high, there are leaks, and so on.

@Rem0o could you please redo it as i would love to try this out ♥

https://github.com/Rem0o/FanControl.ADLX/releases/tag/V0.1


Segmentation fault in simple example

I was going to write a few simple examples. Unfortunately I got a reproduceable segmentation fault for this little example. Happens only in debug mode.
use tracing_subscriber::{prelude::*, EnvFilter};

fn setup_logging() {
    tracing_subscriber::registry()
        .with(tracing_subscriber::fmt::layer().with_writer(std::io::stderr))
        .with(EnvFilter::from_default_env())
        .try_init()
        .ok();
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    setup_logging();
    let mut db = iroh::database::mem::Database::default();
    let rt = iroh::bytes::util::runtime::Handle::from_currrent(1)?;
    let data = [0u8; 1024 * 1024];
    let hash = db.insert(&data);
    let node = iroh::node::Node::builder(db).runtime(&rt).spawn().await?;
    let addrs = node.local_endpoint_addresses().await?;
    println!("Serving {} on {:?}", hash, addrs);
    node.await?;
    Ok(())
}

running in debug mode:
iroh on  iroh-examples [$] via 🦀 v1.71.0 took 2m31s
❯ cargo run --example hello
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/examples/hello`
zsh: segmentation fault  cargo run --example hello

release mode:
iroh on  iroh-examples [$] via 🦀 v1.71.0
❯ cargo run --release --example hello
    Finished release [optimized] target(s) in 0.26s
     Running `target/release/examples/hello`
Serving bafkr4icirxraf5z33f3n4ttqjd2od442o5wynvmcw42i75j36qzltb74va on [<IP_ADDRESS>:49943, <IP_ADDRESS>:49943]

Now here is something scary: it seems to crash before even getting to main... Added this to the beginning of main:
    println!("starting");
    std::io::stdout().flush()?;

...and yet...
❯ cargo run --example hello
   Compiling iroh v0.4.1 (/Users/rklaehn/projects_git/iroh/iroh)
    Finished dev [unoptimized + debuginfo] target(s) in 2.00s
     Running `target/debug/examples/hello`
zsh: segmentation fault  cargo run --example hello

OK, that was easy. I just created a big thing on the stack.
 let data = [0u8; 1024 * 1024];

still a bit weird that you just get a segv...


Support Up-to-date checks (AKA Incremental Build) in Gradle plugin

Enhancement
Gradle plugin doesn't seem to support incremental build => executing generateContractTests task every time, which is slowing down execution of completely not relevant unit tests significantly.
Here is some sample project, I created to simulate our setup, where we have a lot of contract tests (we are using Pact jsons as an input):
https://github.com/anatoliy-balakirev/cloud.contract.demo
Those contract tests in the sample project are not really useful, I just wanted to show the case, where there is a project with a lot of them (in real project we have much more, actually).
So if you get into that project and execute following command a couple of times (it's just a unit test):
./gradlew :test --tests "com.example.demo.service.UserServiceImplTest" --scan
You'll see that generateContractTests is executed every time and takes a couple of seconds even though it's not needed for this particular unit test and its input / output is not changed between runs.
Here is an example of such scan:
https://scans.gradle.com/s/qy3gtncjmopdo/performance/execution
https://scans.gradle.com/s/l2em6bwohe54c/performance/execution
As you can see, there it took almost 2.5 seconds to do something, which is not needed anyway (in our real project it takes up to 10 seconds).
Would be great if plugin could utilise incremental build (check here: https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:up_to_date_checks) and skip tasks if they are up to date.

Great idea @anatoliy-balakirev ! Care for a PR?  What I thought that is working, is not. I will not re-attempt to implement this feature in the upcoming time. If anyone is a Gradle expert, please feel free to try to tackle this problem

@marcingrzejszczak, what's the minimum Gradle version you all are supporting/building against with Hoxton?

We're building against 5.5

Also maintaining a 5.0 compatibility like the Spring Boot Gradle plugin?

I mean assuming that gradle 5.5 is compatible with gradle 5.0 everything should be fine.

To a point, yes. Prime example of where this matters is in the 4.x line. As of 4.8, if memory serves, is where Gradle introduces task configuration avoidance. So in those terms, if we were base lining against a version before that point then the task configuration avoidance API would be off limits. I haven't had a chance to look in depth through the Gradle 5.x API yet, so I'm not sure if Gradle 5.x has any of these types of features that have been introduced during it's lifecycle, but wanted to make sure of the baseline version to ensure feature compatibility.

Hi. Sorry, was quite busy on something else and then was on vacation.
I'm also not so familiar with gradle plugins, but I checked what @marcingrzejszczak did here:
https://github.com/spring-cloud/spring-cloud-contract/commit/31972419e9f95512f97e4affd169cb033f5a3e36
And it seems to be fine, except for one small thing, which is mentioned here:
https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:task_input_output_annotations
Annotations must be placed on getters or on Groovy properties. Annotations placed on setters, or on a Java field without a corresponding annotated getter are ignored.

So when I adjusted ContractsCopyTask to move @InputDirectory from File input to:
	@InputDirectory
	private File getInput() {

and output to:
	@CompileDynamic
	@OutputDirectory
	private File getOutputContractsFolder() {

It alls started to work as expected. I've also modified my sample project to make it easier to use snapshot builds there. So executing my original command with snapshot plugin now gives me this result:
https://scans.gradle.com/s/777u3uwijnjro/performance/execution
11 tasks in 1 project
:compileTestJavaUP-TO-DATE	0.574s
:compileJavaUP-TO-DATE	0.537s
:testUP-TO-DATE	0.076s
:processTestResourcesUP-TO-DATE	0.017s
:copyContractsUP-TO-DATE	0.005s
:compileTestGroovyNO-SOURCE	0.002s
:processResourcesUP-TO-DATE	0.002s
:compileGroovyNO-SOURCE	0.001s
:generateContractTestsUP-TO-DATE	0.001s
:classesUP-TO-DATE	0.000s
:testClassesUP-TO-DATE	0.000s

Which is exactly what we needed! :)
@marcingrzejszczak do you want to proceed with your commit, applying this small fix?

Yeah please do!
I've completely haven't noticed that info about the annotations

Actually, playing with that further it looks like generateContractTests is now always UP-TO-DATE even if I change input for copyContracts one (which is correctly restarted in that case). Will have to check it further.

I mean please feel free to apply a pr, revert my commit that reverted the fix and apply your changes :)

Update on this: I worked on it today and it turned out to be not so easy. There is this global valiable ContractVerifierConfigProperties, which is passed around between tasks, mutated here and there and used as an input for one and output for another. Quite some refactoring is required there, to clean it up and make it more clear what is an input and what is an output to each task. I'm on it.

Yeah, that was exactly the problem :)

So... After a couple of days of refactoring I got something working :)
I've removed ContractVerifierConfigProperties from as many places, as possible. Gradle plugin doesn't use it at all any more. I created dedicated immutable TestGeneratorInput class as an input for TestGenerator, which is now kind of immutable copy of the ContractVerifierConfigProperties. For now I left maven plugin using ContractVerifierConfigProperties, but as soon as it's refactored - ContractVerifierConfigProperties should be removed completely. I decided not to refactor that part within my PR as it's already getting quite big...

https://github.com/spring-cloud/spring-cloud-contract/pull/1165

Unfortunately the current setup has broken the case where someone wants to use the plugin with a git repository. The initContracts task is always up to date, the contracts are never fetched and they are assumed to be there under src/test/resources/contracts, which is wrong.

@marcingrzejszczak I guess they will be downloaded / used the first time and then for all subsequent runs described problem will be the case, right? Do you have some sample config to reproduce this?

Here is the sample https://github.com/spring-cloud-samples/spring-cloud-contract-samples/tree/2.2.x/producer_with_git
Unfotunately there are the following issues

the initTask is never called since it's always up to date
even if I make it not always uptodate, what happens is that

if (downloaded != null) {
            logger.info("Downloaded contracts to [" + downloaded + "]")
            config.includedContracts.set(downloaded.inclusionProperties.includedContracts)
            config.includedRootFolderAntPattern.set(downloaded.inclusionProperties.includedRootFolderAntPattern)
            config.initialisedContractsDirectory.set(downloaded.downloadedContracts)
        }
        logger.info("For project [{}] will use contracts provided in the folder [{}]", project.name, config.initialisedContractsDirectory.get())

running this ^^ results in
Downloaded contracts to [org.springframework.cloud.contract.verifier.plugin.GradleContractsDownloader$DownloadedData(/tmp/git-contracts-1566818849546-0/META-INF/com.example/beer-api-producer-git/0.0.1.BUILD-SNAPSHOT/contracts, org.springframework.cloud.contract.stubrunner.ContractDownloader$InclusionProperties@751f17e7)]
For project [beer-api-producer-git] will use contracts provided in the folder [/home/marcin/repo/spring-cloud-contract-samples/producer_with_git/src/test/resources/contracts]

as you can see, even though we set a configuration element, the get() method returns sth else. I fixed that, but there's no option to make one configuration change the output and pass it to another task.

as you can see, even though we set a configuration element, the get() method returns sth else.
That's quite weird...
I fixed that,
Just out of curiosity, how exactly did you fix that? Just wonder if that fix actually caused this:
but there's no option to make one configuration change the output and pass it to another task.
Updating the property should be propagated everywhere, in theory... That output property is used as an input to other tasks.

Just out of curiosity, how exactly did you fix that? Just wonder if that fix actually caused this:

afair I've removed some @Optional annotations.
Unfortunately, due to the amount of time I have to put into Gradle, and Gradle's lack of being developer friendly in any way, I will timebox it to a day or two and most likely will be forced to remove this feature or add some hacks to make the tests pass again.

I can take a look into this. Maybe whole idea to try to make that init part incremental was wrong because there is already caching in the GradleContractsDownloader...

Yeah looks like it

By when do you need this fixed?

The sooner, the better ;)

Trying to run that project locally and getting:
Caused by: org.eclipse.jgit.errors.NoRemoteRepositoryException: file:///Users/anatolii.balakiriev/Developer/spring-cloud-contract-samples/producer_with_git/build/contract_git/.git: not found.

Seems to be caused by this config:
	contractRepository {
		repositoryUrl = "git://file://${project.buildDir}/contract_git/"
	}

Any suggestions on how to make it working?

It works on my machine :tm:
just copy the contract_git folder from root of the project to build/contract_git and mv build/contract_git/git -> build/contract_git/.git. A Gradle task there should do that.
$ mkdir build
$ cp -r ../contract_git build/contract_git
$ mv build/contract_git/git build/contract_git/.git

Ok, thx. There actually seems to be a gradle task to do that, but it has a wrong dependency on:
copyContracts.dependsOn("prepareGit")

and must be this instead:
initContracts.dependsOn("prepareGit")

So after applying that and disabling incremental build for initContracts task like this:
		// TODO: Enable it when all caching from `GradleContractsDownloader` is replaced with Gradle's one here
		// @OutputDirectory
		DirectoryProperty initialisedContractsDirectory

I got another problem in the generateContractTests task. It looks like there was a bug there, which was ignoring inclusion pattern, generated by the GradleContractsDownloader, so that task was always including everything. Here is an old log (the last line is an interesting one here):
> Task :generateContractTests
Task ':generateContractTests' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
Generated test sources dir [/Users/anatolii.balakiriev/Developer/spring-cloud-contract-samples/producer_with_git/build/generated-test-sources/contracts]
Spring Cloud Contract Verifier Plugin: Invoking test sources generation
Contracts are unpacked to [/Users/anatolii.balakiriev/Developer/spring-cloud-contract-samples/producer_with_git/build/stubs/META-INF/com.example/beer-api-producer-git/0.0.1.BUILD-SNAPSHOT/contracts]
Included contracts are [.*]

And after I started propagating that properly - we have this log:
> Task :generateContractTests
Caching disabled for task ':generateContractTests' because:
  Build cache is disabled
Task ':generateContractTests' is not up-to-date because:
  Output property 'config.generatedTestResourcesDir' file /Users/anatolii.balakiriev/Developer/spring-cloud-contract-samples/producer_with_git/build/generated-test-resources/contracts has been removed.
  Output property 'config.generatedTestSourcesDir' file /Users/anatolii.balakiriev/Developer/spring-cloud-contract-samples/producer_with_git/build/generated-test-sources/contracts has been removed.
Generated test sources dir [/Users/anatolii.balakiriev/Developer/spring-cloud-contract-samples/producer_with_git/build/generated-test-sources/contracts]
Generated test resources dir [/Users/anatolii.balakiriev/Developer/spring-cloud-contract-samples/producer_with_git/build/generated-test-resources/contracts]
Spring Cloud Contract Verifier Plugin: Invoking test sources generation
Contracts are unpacked to [/Users/anatolii.balakiriev/Developer/spring-cloud-contract-samples/producer_with_git/build/stubs/META-INF/com.example/beer-api-producer-git/0.0.1.BUILD-SNAPSHOT/contracts]
Included contracts are [^/var/folders/54/71d7df0530ggpmhmbp_0_dv40000gn/T/git-contracts-1566891881312-0/META-INF/com.example/beer-api-producer-git/0.0.1.BUILD-SNAPSHOT/contracts.*$]

So as you can see, now we have correct includedContracts, but in fact it doesn't make sense here, as those should already be copied and that temp directory doesn't matter...

Here is an old code, which I actually remember we even discussed:
		if (getConfigProperties().getContractDependency()) {
			project.logger.debug("Updating the stubs locations for the case where we have a JAR with contracts")
			props.contractsDslDir = contractsDslDir
			props.includedContracts = ".*"
		}

So it was always setting it to ".*" for some reason (as getConfigProperties().getContractDependency() could never be null). I'm not sure what was the logic around it, but will just restore that behaviour for now.

and must be this instead:

we also check for backward compatibility in those samples, so most likely it will fail cause Finchley doesn't have the initContracts task. Don't worry about it, I'll fix it later.

Here is an old code, which I actually remember we even discussed:

Ah yeah, you're right, now I remember.

So it was always setting it to ".*" for some reason (as getConfigProperties().getContractDependency() could never be null). I'm not sure what was the logic around it, but will just restore that behaviour for now.

Yeah, restore it for now. We'll deal with that later.

Created PR. Once that one is merged - I can also try to take a look into replacing custom caching in the GradleContractsDownloader with Gradle's incremental build. If we do that - we won't even need InitContractsTask any more (was created to be kind of a bridge between that custom caching and Gradle's incremental build).

I just checked that further and today caching of downloaded artifacts is done in a way that they are cached per groupId / artifactId. I.e. if caching is enabled and I run plugin a couple of times - we'll get following behaviour:

groupId:1, artifactId:1 -> downloaded
groupId:1, artifactId:1 -> using cached from step #1
groupId:2, artifactId:2 -> downloaded
groupId:1, artifactId:1 -> using cached from step #1
groupId:2, artifactId:2 -> using cached from step #3

With Gradle's caching, steps #4 and #5 will trigger new download, as input was changed (i.e. it won't remember history of executions, only the previous one). So will be:

groupId:1, artifactId:1 -> downloaded
groupId:1, artifactId:1 -> using cached from step #1
groupId:2, artifactId:2 -> downloaded
groupId:1, artifactId:1 -> downloaded
groupId:2, artifactId:2 -> downloaded

Is that ok?

I guess so. However, with the history of working with Gradle I really feel scared to rely on anything that Gradle provides. We have no certainty that with Gradle 25.0 that with their current pace might be released in a month this feature will not be removed. OTOH it's not a gigantic change I guess so yeah, go ahead with using Gradle caching.

:) hopefully won't be removed.
Ok, will try to do that.


Prevent the optimizer from selecting an index based on ORDER BY startedAt.

🚢

I did add some utility methods for checking if running Postgres or MySQL in one of the classss. A hack would be to do a conditional there

It should be compatible; this change is equivalent to sorting by a function, which both MySQL and Postgres support. I was originally going to go with an index hint, but that definitely wouldn't have been compatible on the Postgres side.
For what it's worth, we're not intending to merge this change into master.

@baconmania where are we going with this PR, leaving it in this branch for now or do we have a version we could merge to master?

@baconmania Since we've found the issue internally, should we update this PR to instead be a poller that occasionally runs a table stats update? Thinking update rather than just closing since it'd be a pain to unmerge these commits from the branches

We're handling this internally in a different way now, so I'm closing this.


issue with cache_options in 0.6.0 using intake.upload with azure datalake

Traceback (most recent call last):
  File "/home/vsts/work/1/s/machine_readable_spec/transformations/push_to_datalake.py", line 38, in <module>
    entry = intake.upload([infile_read], outloc_latest, storage_options=global_storage_options)
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/intake/container/__init__.py", line 54, in upload
    s = cls._data_to_source(data, path, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/intake/container/semistructured.py", line 95, in _data_to_source
    dask.compute(out)
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/dask/base.py", line 436, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/dask/threaded.py", line 81, in get
    **kwargs
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/dask/core.py", line 119, in _execute_task
    return func(*args2)
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/intake/container/semistructured.py", line 101, in write_file
    with fo as f:
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/fsspec/core.py", line 98, in __enter__
    f = self.fs.open(self.path, mode=mode)
  File "/opt/hostedtoolcache/Python/3.7.5/x64/lib/python3.7/site-packages/fsspec/spec.py", line 725, in open
    **kwargs
TypeError: _open() got an unexpected keyword argument 'cache_options'
##[error]/opt/hostedtoolcache/Python/3.7.5/x64/bin/python failed with return code: 1

pip freeze:
adal==1.2.2
adlfs==0.1.0
appdirs==1.4.3
attrs==19.3.0
azure-common==1.1.23
azure-datalake-store==0.0.48
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
boltons==19.3.0
certifi==2019.9.11
cffi==1.13.2
chardet==3.0.4
cloudpickle==1.2.2
cryptography==2.8
dask==2.7.0
entrypoints==0.3
fsspec==0.6.0
idna==2.8
importlib-metadata==0.23
intake==0.5.3+36.gba252a0
Jinja2==2.10.3
jsonschema==3.1.1
locket==0.2.0
MarkupSafe==1.1.1
more-itertools==7.2.0
msgpack==0.6.2
numpy==1.17.4
pandas==0.25.3
partd==1.0.0
pyarrow==0.15.1
pycparser==2.19
PyJWT==1.7.1
pyrsistent==0.15.5
python-dateutil==2.8.1
pytz==2019.3
PyYAML==5.1.2
requests==2.22.0
six==1.13.0
toolz==0.10.0
urllib3==1.25.7
zipp==0.6.0
I'm not sure what other things I can add as extra info, if there is anything, please let me know

For clarity: it worked with the same installs & code except for fsspec==0.5.2

This was caused by missing **kwargs in the _open method for adlfs, which should be in the current version.  I encountered it when using the distributed library.


Horizontal bar chart legend padding & sort order regression in 4.0.2rc2

Bug description
Testing out https://github.com/apache/superset/pull/29345 and it introduces a regression on a chart vs. 4.0.0.  The top bar of my chart now goes over the legend and its axis label is not visible.
It also seems to have reversed the sorting order.
How to reproduce the bug
Create bar chart and switch on horizontal x-axis.  If there are other aspects of my chart that are relevant to the bug I am happy to hop on a Zoom or chat to try to isolate that.
Screenshots/recordings
Before (4.0.0):

4.0.2rc2:

Superset version
master / latest-dev
Python version
3.9
Node version
16
Browser
Chrome
Additional context
No response
Checklist

[ ] I have searched Superset docs and Slack and didn't find a solution to my problem.
[ ] I have searched the GitHub issue tracker and didn't find a similar bug report.
[ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

It also seems to have reversed the sorting order.

@sfirke this is expected as described in https://github.com/apache/superset/pull/29345:

This will change the default y-axis order of horizontal bar charts.

About the problem with the bar that goes over the legend, it looks like it happens in some specific combination of the number of bars and labels length. It's an ECharts problem so we would need a fix there to consistently fix this problem for all charts. @villebro any idea?
Regarding Superset releases, I think the current state is still better than the state before the reverts. You can check the screenshots of https://github.com/apache/superset/pull/29345 to see the problems.
So, my vote would be to proceed with the release and open a bug in ECharts for this problem.

Wow, I had added a legend margin of -75 in my test instance from 4.0.2rc1 when I was seeing how I would work around this. Then I didn't reset the metadata DB for testing RC2.  This is just fine now.
And re: the sorting, that's fine with me, I had missed that sentence in the PR.  I just unchecked the box Y-Axis Sort Ascending and I get back to my original order under 4.0.0.  That's easy enough.
Sorry for the false alarm issue, thank you for the quick reply and investigation.  I will close this.

😅


Import referenced headers (i.e., nested imports)

The implementation will depend on whether u-ctags supports or will support this itself.

This will make the pointer fallback for type detection obsolete.


StateError thrown when accessing the export namespace of a library element

From crash reporting:
Thread 0 [MAIN THREAD]
StateError: Bad state: Expected existing element: <path>
  at LinkedElementFactory.elementOfReference	(linked_element_factory.dart:209)
  at LinkedElementFactory.elementOfReference	(linked_element_factory.dart:198)
  at LinkedElementFactory.buildExportNamespace	(linked_element_factory.dart:91)
  at LibraryElementImpl.exportNamespace	(element.dart:4205)
  at LibraryElementExtensions.exportedExtensions	(element.dart:61)
  at _ImportLibraryContainingExtension.compute	(import_library.dart:459)
  at FixProcessor._addFromProducers.compute	(fix_internal.dart:1794)
  at FixProcessor._addFromProducers	(fix_internal.dart:1820)
  at <asynchronous gap>	(async)
  at FixProcessor.compute	(fix_internal.dart:1749)
  at <asynchronous gap>	(async)
  at DartFixContributor.computeFixes	(fix_internal.dart:257)
  at <asynchronous gap>	(async)
  at DartCodeActionsProducer.getFixActions	(dart.dart:172)
  at <asynchronous gap>	(async)
  at OperationPerformanceImpl.runAsync	(operation_performance.dart:172)
  at <asynchronous gap>	(async)
  at CodeActionHandler.handle	(handler_code_actions.dart:202)
  at <asynchronous gap>	(async)
  at ServerStateMessageHandler.handleMessage	(handlers.dart:287)
  at <asynchronous gap>	(async)
  at LspAnalysisServer._handleRequestMessage	(lsp_analysis_server.dart:994)
  at <asynchronous gap>	(async)
  at LspAnalysisServer.handleMessage.<fn>.<fn>	(lsp_analysis_server.dart:471)
  at <asynchronous gap>	(async)
  at OperationPerformanceImpl.runAsync	(operation_performance.dart:172)
  at <asynchronous gap>	(async)
  at LspAnalysisServer.handleMessage.<fn>	(lsp_analysis_server.dart:454)

There's a second report that might be related:
Thread 0 [MAIN THREAD]
_TypeError: Null check operator used on a null value
  at LibraryElementImpl.exportNamespace	(element.dart:4177)
  at NotImportedContributor.computeSuggestions	(not_imported_contributor.dart:96)
  at <asynchronous gap>	(async)
  at DartCompletionManager.computeSuggestions.<fn>	(completion_manager.dart:195)
  at <asynchronous gap>	(async)
  at OperationPerformanceImpl.runAsync	(operation_performance.dart:172)
  at <asynchronous gap>	(async)
  at DartCompletionManager.computeSuggestions	(completion_manager.dart:192)
  at <asynchronous gap>	(async)
  at CompletionHandler._getServerDartItems.<fn>	(handler_completion.dart:393)
  at <asynchronous gap>	(async)
  at OperationPerformanceImpl.runAsync	(operation_performance.dart:172)
  at <asynchronous gap>	(async)
  at CompletionHandler._getServerDartItems	(handler_completion.dart:387)
  at <asynchronous gap>	(async)
  at CompletionHandler.handle.<fn>	(handler_completion.dart:166)
  at <asynchronous gap>	(async)
  at OperationPerformanceImpl.runAsync	(operation_performance.dart:172)
  at <asynchronous gap>	(async)
  at CompletionHandler.handle	(handler_completion.dart:204)
  at <asynchronous gap>	(async)
  at ServerStateMessageHandler.handleMessage	(handlers.dart:352)
  at <asynchronous gap>	(async)
  at LspAnalysisServer._handleRequestMessage	(lsp_analysis_server.dart:1005)
  at <asynchronous gap>	(async)
  at LspAnalysisServer.handleMessage.<fn>.<fn>	(lsp_analysis_server.dart:471)
  at <asynchronous gap>	(async)
  at OperationPerformanceImpl.runAsync	(operation_performance.dart:172)
  at <asynchronous gap>	(async)
  at LspAnalysisServer.handleMessage.<fn>	(lsp_analysis_server.dart:454)

If the two are not related, please open a separate issue for the second.


Fixing Autoscaling

Work in progress

Trying to make the autoscaling working for a custom metric based on requests per second.
I'm trying to configure properly Prometheus Operator & Adapter in order to be able to have a HorizontalPodAutoscaler.
At the moment I get this:
cloud-phoenix-kata git:(fix/autoscaling)$ kubectl get -n cloud-phoenix-kata hpa
NAMESPACE            NAME            REFERENCE                  TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
cloud-phoenix-kata   endpoint-node   Deployment/endpoint-node   <unknown>/10   1         10        3          31m

cloud-phoenix-kata git:(fix/autoscaling)$ kubectl describe -n cloud-phoenix-kata hpa endpoint-node
Name:                             endpoint-node
Namespace:                        cloud-phoenix-kata
Labels:                           <none>
Annotations:                      kubectl.kubernetes.io/last-applied-configuration:
                                    {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"endpoint-node","namespace":"clou...
CreationTimestamp:                Wed, 13 Nov 2019 21:00:01 +0100
Reference:                        Deployment/endpoint-node
Metrics:                          ( current / target )
  "requests-per-second" on pods:  <unknown> / 10
Min replicas:                     1
Max replicas:                     10
Deployment pods:                  3 current / 0 desired
Conditions:
  Type           Status  Reason               Message
  ----           ------  ------               -------
  AbleToScale    True    SucceededGetScale    the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetPodsMetric  the HPA was unable to compute the replica count: unable to get metric requests-per-second: unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
Events:
  Type     Reason                        Age                      From                       Message
  ----     ------                        ----                     ----                       -------
  Warning  FailedGetObjectMetric         4h46m (x484 over 31m)  horizontal-pod-autoscaler  unable to get metric requests-per-second: Ingress on cloud-phoenix-kata main-route/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
  Warning  FailedComputeMetricsReplicas  4h15m (x12 over 18m)   horizontal-pod-autoscaler  Invalid metrics (1 invalid out of 1), last error was: failed to get object metric value: unable to get metric requests-per-second: Ingress on cloud-phoenix-kata main-route/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
  Warning  FailedGetObjectMetric         4h8m (x41 over 18m)    horizontal-pod-autoscaler  unable to get metric requests-per-second: Ingress on cloud-phoenix-kata main-route/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
  Warning  FailedComputeMetricsReplicas  4h3m (x12 over 6m)     horizontal-pod-autoscaler  Invalid metrics (1 invalid out of 1), last error was: failed to get object metric value: unable to get metric requests-per-second: Ingress on cloud-phoenix-kata main-route/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
  Warning  FailedGetObjectMetric         6m33s (x60 over 6m)    horizontal-pod-autoscaler  unable to get metric requests-per-second: Ingress on cloud-phoenix-kata main-route/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
  Warning  FailedGetPodsMetric           10s (x3 over 43s)        horizontal-pod-autoscaler  unable to get metric requests-per-second: unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered
  Warning  FailedComputeMetricsReplicas  10s (x3 over 42s)        horizontal-pod-autoscaler  Invalid metrics (1 invalid out of 1), last error was: failed to get object metric value: unable to get metric requests-per-second: unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered