Keep up to date with changes in the engine regarding TTS configuration

https://github.com/daisy/pipeline-modules/issues/66

This will require backend changes only: this will have to be specified either at launch time, or through the settings API (like is already the case for other settings such as speech-rate)
https://daisy-dev.slack.com/archives/C064GB8U9/p1723476887384829?thread_ts=1723476120.081709&cid=C064GB8U9

Setting the "org.daisy.pipeline.tts.config" property at launch time is already possible and will be the most straightforward solution, and should give us exactly the same behavior.
Setting the property dynamically through the settings API is not possible yet today, but I'm about to change that. The main benefit of this approach is that changing a property affects the jobs created after the change, but not the jobs created before the change (but still in the cueue). So through the dynamic approach, you could in theory create a new TTS configuration file after each change.

Instead of using a script option to point to a TTS configuration file, the TTS configuration file can be passed at launch time via the org.daisy.pipeline.tts.config property.

@bertfrees will the scripts still have a tts config option? Not that we would use it. But I need to know if I still need to filter it out on the script form.

I will keep it around for now. An alternative the the script option is also still needed for the command line interface.


subsys: tracing: add support for tracing power

This adds support for tracing power management subsystem.

see #35340

see #35340

Great. This commit also adds CTF support. You may wanna add definitions for this in your PR as well.


Explore switching to bastion service

The bastion service is now GA: https://blogs.oracle.com/cloudsecurity/post/secure-access-with-oci-bastion
Explore the bastion module to the bastion service:

Document changes in policy required
What happens when the session expires and a timeout happens or there's a subsequent run of terraform apply?
Provide configuration option to use bastion service or use a compute instead. Use conditional module?
Impact on downstream projects such as OKE especially when they use null_resource

This can be done in downstream modules. We'll keep using bastion host for IaC purposes


Request RC32 on the download page

We have issues on performance, all our servers be loaded caused the Influxdb eat I/0 CPU

It seems that there are a lot of improvements on the RC32, is it possible to get it compiled in the deb package ?

@dtouzeau rc32 will likely be cut sometime today. Please test this again once rc32 is out and open a new bug report if you continue to run into performance problems.


 User uploaded images are need to be put in assets folder instead of public/uploads as we did in Angular 1.
I have updated path on widget.service.server.

Updated code in pull request #22


Update

This update is required to allow the Kolmogorov-Smirnov tests to be run correctly on different versions of scipy.


added has many query

I went ahead and added a has many query for this, which should be good to go! I peaked through the tests a little, but I wasn't sure how exactly you had them setup. If you could instruct a little bit on how to add a test for this within your setup then I can go ahead and do that too!

I was working through something on my project, and I found that I needed direct access to the performJoin function. The issue I came into was that using the orWhere function within a nested where did not affect the outer join. The issue with this is that it does not allow you to have grouped wheres such as...
SELECT foo.* JOIN bar ON foo.id = bar.foo_id WHERE foo.column_1 = 'something' AND (foo.column_2 LIKE '%a different thing%' OR foo.bar.a_different_column > 5)

Previously the nested wheres in the or section would be built, but the outer join would get lost.

I have different ideas for HasMany relation and it will be implemented soon. You can use your fork for your needs if you want...

Do you have any instructions one that?


most-used | less dies with a weird error

$ most-used | less
[prints 27 lines]
lines 1-27zsh: suspended (tty input)  most-used | less

Actually, it backgrounds itself and must be resuscitated with fg. Super weird!


Add documentation on unit testing

Initiative / goal
While LitElement docs are clear, they don't provide any guide on how to test components.
People have to discover https://open-wc.org for some tools & recommendations.
But most of newbie developers are unlikely to find it quickly.
In contrast, Stencil docs have a separate section on testing.
Scope

Improving docs to add testing guide

Acceptance criteria and must have scope

Developers should be able to setup unit tests  for their components with karma and mocha without having to search for any external documentation and resources.

The new docs site now covers unit testing with recommendations on test frameworks and example configs: https://lit.dev/docs/tools/testing/


[DOC] Manual version is "pre-git"

In the manual here- https://nixos.org/nixpkgs/manual/ version is Version 20.03pre-git . Minor but figured you guys might like to know.
cc: @worldofpeace @disassembler

https://github.com/NixOS/nixos-homepage/issues/417


”Zorlențu Mare”, ”Zorlencior” are listed in the drop-down menu

Description:
When user writes ”Zorlențu Mare”, ”Zorlencior” the locations appear in the drop-down menu next to the comune and the county witch is a part according to the law.
Precondition:
The website is up an running.

Step 1
Write in the search bar "Zorlențu Mare"

Expected results
The location appears in the drop-down menu as "Sat Zorlențu Mare, CARAȘ-SEVERIN(Zorlențu Mare)".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Zorlencior"

Expected results
The location appears in the drop-down menu as "Sat Zorlencior, CARAȘ-SEVERIN(Zorlențu Mare)".


Integer property with large min/max value fails

Description
A property like the following
seed:
  type: integer
  minimum: -9223372036854776000
  maximum:<PHONE_NUMBER>854776000
  nullable: true
  description: >

fails with Expected an Integer literal but found a floating point value (9.223372036854776e+18)
error: Inconsistency encountered when parsing `maximum` in Document.components.schemas.CreateChatCompletionRequest.properties.seed: Expected an Integer literal but found a floating point value (9.223372036854776e+18).Swift OpenAPI Generator is running with the following configuration:

- OpenAPI document path: /Sources/OpenAIAsyncHTTPClient/openapi.yaml
- Configuration path: /Sources/OpenAIAsyncHTTPClient/openapi-generator-config.yaml
- Generator modes: types, client
- Access modifier: public
- Feature flags: <none>
- Output file names: Types.swift, Client.swift
- Output directory: /Sources/OpenAIAsyncHTTPClient/GeneratedSources
- Diagnostics output path: <none - logs to stderr>
- Current directory: /Users/atacan/Developer/Repositories/SwiftOpenAIAPI
- Plugin source: command
- Is dry run: false
- Additional imports: <none>

Reproduction
You can use https://github.com/openai/openai-openapi/blob/c880fa100c99e42c029ef8497cb97aa63a8ab706/openapi.yaml#L12044
and manually generate swift package generate-code-from-openapi
Package version(s)
.
├──<EMAIL_ADDRESS>│   ├──<EMAIL_ADDRESS>│   │   └──<EMAIL_ADDRESS>│   ├──<EMAIL_ADDRESS>│   │   └──<EMAIL_ADDRESS>│   ├──<EMAIL_ADDRESS>│   └──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>│   └──<EMAIL_ADDRESS>├──<EMAIL_ADDRESS>│   ├──<EMAIL_ADDRESS>│   │   └──<EMAIL_ADDRESS>│   ├──<EMAIL_ADDRESS>│   └──<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>    ├──<EMAIL_ADDRESS>    │   ├──<EMAIL_ADDRESS>    │   ├──<EMAIL_ADDRESS>    │   └──<EMAIL_ADDRESS>    ├──<EMAIL_ADDRESS>    │   ├──<EMAIL_ADDRESS>    │   │   ├──<EMAIL_ADDRESS>    │   │   ├──<EMAIL_ADDRESS>    │   │   └──<EMAIL_ADDRESS>    │   ├──<EMAIL_ADDRESS>    │   │   └──<EMAIL_ADDRESS>    │   │       ├──<EMAIL_ADDRESS>    │   │       ├──<EMAIL_ADDRESS>    │   │       └──<EMAIL_ADDRESS>    │   ├──<EMAIL_ADDRESS>    │   │   ├──<EMAIL_ADDRESS>    │   │   │   ├──<EMAIL_ADDRESS>    │   │   │   ├──<EMAIL_ADDRESS>    │   │   │   └──<EMAIL_ADDRESS>    │   │   └──<EMAIL_ADDRESS>    │   ├──<EMAIL_ADDRESS>    │   │   ├──<EMAIL_ADDRESS>    │   │   │   ├──<EMAIL_ADDRESS>    │   │   │   ├──<EMAIL_ADDRESS>    │   │   │   └──<EMAIL_ADDRESS>    │   │   ├──<EMAIL_ADDRESS>    │   │   │   ├──<EMAIL_ADDRESS>    │   │   │   │   ├──<EMAIL_ADDRESS>    │   │   │   │   ├──<EMAIL_ADDRESS>    │   │   │   │   └──<EMAIL_ADDRESS>    │   │   │   └──<EMAIL_ADDRESS>    │   │   └──<EMAIL_ADDRESS>    │   ├──<EMAIL_ADDRESS>    │   │   ├──<EMAIL_ADDRESS>    │   │   │   ├──<EMAIL_ADDRESS>    │   │   │   ├──<EMAIL_ADDRESS>    │   │   │   └──<EMAIL_ADDRESS>    │   │   └──<EMAIL_ADDRESS>    │   ├──<EMAIL_ADDRESS>    │   ├──<EMAIL_ADDRESS>    │   └──<EMAIL_ADDRESS>    │       └──<EMAIL_ADDRESS>    ├──<EMAIL_ADDRESS>    │   └──<EMAIL_ADDRESS>    └──<EMAIL_ADDRESS>
Expected behavior
parse it as integer
Environment
swift-driver version: 1.115 Apple Swift version 6.0 (swiftlang-<IP_ADDRESS>.10 clang-16<IP_ADDRESS>)
Target: arm64-apple-macosx15.0

Additional information
No response

Int.max is<PHONE_NUMBER>854775807, but the value in the document is higher, so it fails to parse (ends in 776000).
So the failure seems expected.
Who wrote the OpenAPI document? Why does it have a value larger than Int.max (on 64-bit platforms)?

Thank you for the information.

Who wrote the OpenAPI document?

It's OpenAI's official document
https://github.com/openai/openai-openapi/blob/c880fa100c99e42c029ef8497cb97aa63a8ab706/openapi.yaml#L12044

This is not the first inaccuracy I've seen in this OpenAPI document. Perhaps you can file an issue on their repo? https://github.com/openai/openai-openapi


Clarification for Lr Scheduler ReduceLROnPlateau in PL

Maybe I missed it in PL documentation; however, when using the learning rate scheduler " ReduceLROnPlateau" appears to take an argument for which metric to monitor during training.
Does PL handle passing this argument? If so is there anyway for the user to specify which argument is passed to the scheduler?
From Pytorch Docs, example shows scheduler.step takes val_loss ,as the argument for when to reduce learning rate for this particular scheduler.
Example

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min')
for epoch in range(10):
train(...)
val_loss = validate(...)
# Note that step should be called after validate()
scheduler.step(val_loss)

After triple checking documentation, I found a NOTE that I missed initially, which address this issue.
Sorry to bother.


FAIL: //tensorflow/core:common_runtime_direct_session_with_tracking_alloc_test

System Info
  Name:						 Hawaii
  Vendor:					 Advanced Micro Devices, Inc.
  Device OpenCL C version:			 OpenCL C 2.0
  Driver version:				 1912.5 (VM)
  Profile:					 FULL_PROFILE
  Version:					 OpenCL 2.0 AMD-APP (1912.5)
  Extensions:					 cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_gl_sharing cl_khr_gl_depth_images cl_ext_atomic_counters_32 cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_khr_spir cl_khr_subgroups cl_khr_gl_event cl_khr_depth_images cl_khr_mipmap_image cl_khr_mipmap_image_writes

ComputeCpp 0.2.0
To reproduce
bazel test --config=sycl --local_test_jobs=4 -k --test_lang_filters=cc,py --action_env=LD_PRELOAD=/usr/lib/libOpenCL.so.1 --test_timeout 300,750,1200,3600 //tensorflow/core:common_runtime_direct_session_with_tracking_alloc_test

Error
[ RUN      ] DirectSessionWithTrackingAllocTest.CostModelWarmup
2017-06-12 15:57:58.363361: F ./tensorflow/core/common_runtime/direct_session_with_tracking_alloc_test.cc:141] Check failed: cost_models.size() == 1 (2 vs. 1)
external/bazel_tools/tools/test/test-setup.sh: line 159: 23310 Aborted                 (core dumped) "${TEST_PATH}" "$@"

That seems to be failing for CUDA as well:
2017-06-15 15:17:13.127249: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 15:17:13.162545: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 15:17:13.162550: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-15 15:17:13.300625: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-06-15 15:17:13.300933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:938] Found device 0 with properties:
name: GeForce GTX 980
major: 5 minor: 2 memoryClockRate (GHz) 1.2785
pciBusID 0000:01:00.0
Total memory: 3.94GiB
Free memory: 3.87GiB
2017-06-15 15:17:13.300946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:959] DMA: 0
2017-06-15 15:17:13.300950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:969] 0:   Y
2017-06-15 15:17:13.300960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1028] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0)
[       OK ] DirectSessionWithTrackingAllocTest.CostModelTest (204 ms)
[ RUN      ] DirectSessionWithTrackingAllocTest.CostModelWarmup
2017-06-15 15:17:13.320060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1028] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0)
2017-06-15 15:17:13.324450: F tensorflow/core/common_runtime/direct_session_with_tracking_alloc_test.cc:141] Check failed: cost_models.size() == 1 (2 vs. 1)
external/bazel_tools/tools/test/test-setup.sh: line 159:   333 Aborted                 (core dumped) "${TEST_PATH}" "$@"

Reducing priority.

Fixed by #114 and #115

This test is marked no_gpu, so we shouldn't be running it on SYCL anyway. The test behaviour changed in #115 may actually be intended, so the test fails when run with any devices other than CPU.

Closing


License for repo

Hi Yuan. Thanks for your work on ReMASC! Just trying to get a baseline to work from and wanted to run your network here against a split of the data. Would it be possible to add a license to the repository, if you are happy for it to be used freely?
Thanks!

thanks!


Enemies need to be over villagers for sprites (e.g. flies, snakes)? This may look weird but otherwise the villager can hide things that kill you. Alternately we could have the villagers leave the screen quickly.Thoughts?

@esalling23 I think we do need to have the flies / snakes go above the villgaers. Play level2.2. It's hard with the flies under the villagers.

@wadek2 There is now a "Villager" sprite layer for villagers - Flies etc. are still "Obstacles" and Villagers appear below that layer. Just assign the villagers to the Villager layer in their sprite renderer


Iteration.Root() no longer returns an iteration if it doesn't actually exist.

Please describe what your change is about.
This PR updated the validation mechanism (and all related tests) so that the iteration.Root method doesn't return an iteration if it doesn't actually exist.

What issues does it solve?
https://github.com/fabric8-services/fabric8-wit/issues/2122

Codecov Report

Merging #2126 into master will increase coverage by 0.03%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master    #2126      +/-   ##
==========================================
+ Coverage   69.02%   69.06%   +0.03%
==========================================
  Files         162      162
  Lines       15140    15144       +4
==========================================
+ Hits        10451    10459       +8
+ Misses       3738     3735       -3
+ Partials      951      950       -1

Impacted Files
Coverage Δ

controller/workitem.go
78.64% <0%> (ø)
:arrow_up:

iteration/iteration.go
66.85% <66.66%> (+3%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14228fc...61a257b. Read the comment docs.

[test]


Erroneously inserted rows with  statement pooling enabled and when a Connection gets re-used from the ConnectionPool

Driver version

com.microsoft.sqlserver
mssql-jdbc
12.6.1.jre11

SQL Server version
Microsoft SQL Server 2019 (RTM-CU18) (KB5017593) - 15.0.4261.1 (X64)
	Sep 12 2022 15:07:06
	Copyright (C) 2019 Microsoft Corporation
	Developer Edition (64-bit) on Windows 10 Pro 10.0 <X64> (Build 19045: )

Client Operating System
Windows 10 Pro
JAVA/JVM version
Java 17
Problem description
There are some duplicate inserts and some missing entries when a custom connection pool is being utilized and server side statement pooling is enabled (using the setDisableStatementPooling flag). When the connection is re-used from the connection pool and the flag setDisableStatementPooling is set to false this issue seems to happen.
To test this instance, I was able to simulate a very basic stand alone as below :
Please see the following test program, where I intentionally return the previously created connection object from the _userToConnectionsMap  cache map for subsequent runs.
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Collection;
import java.util.Iterator;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.locks.ReentrantLock;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.sql.PooledConnection;

public class SQLConnectionPool {

	 final private ReentrantLock _connLock = new ReentrantLock();
	 final private Map _userToConnectionsMap = new ConcurrentHashMap();
	 static String tableName = "batchTest";
	/**
	 * Crate a new connection
	 *
	 * @return the new created connection
	 * @throws SQLException When fail to create a new connection.
	 */
	private Connection createNewConnection(final String username, final String password) throws SQLException {

		com.microsoft.sqlserver.jdbc.SQLServerConnectionPoolDataSource msDs = new com.microsoft.sqlserver.jdbc.SQLServerConnectionPoolDataSource();
		msDs.setUser(username);
		msDs.setApplicationName("SQL Server Duplicate Row");
		msDs.setPassword(password);
		msDs.setServerName("<>");
		msDs.setPortNumber(0);
		msDs.setDatabaseName("<>");
		msDs.setInstanceName("<>");
		msDs.setEncrypt("false");
		msDs.setDisableStatementPooling(false);
		msDs.setStatementPoolingCacheSize(20);
		msDs.setServerPreparedStatementDiscardThreshold(20);
		msDs.setSendTemporalDataTypesAsStringForBulkCopy(false);
		// ... create the ACTUAL new connection...
		PooledConnection actual = msDs.getPooledConnection(username, password);
		// ... cache it...
		Collection conns = (Collection) _userToConnectionsMap.get(username);
		if (conns == null || conns.isEmpty()) {
			conns = new CopyOnWriteArrayList();
			_userToConnectionsMap.put(username, conns);
		}

		conns.add(actual);
		return actual.getConnection();
	}

	final public Connection getConnection(final String username, final String password)
			throws SQLException, InterruptedException {

		if (username == null) {
			throw new NullPointerException("The username cannot be null.");
		}
		if (password == null) {
			throw new NullPointerException("The password cannot be null.");
		}

		// Acquire a lock...
		_connLock.lock();
		// Declare the connection to be returned...
		Connection conn = null;
		try {

			conn = getAvailableConnection(username);
			// If none found...
			if (conn == null) {
				conn = createNewConnection(username, password);
			}
//
		} finally {
			// Ensure that the lock is released...
			_connLock.unlock();
		}
		// Return the connection...
		return conn;
	}

	private Connection getAvailableConnection(final String username) throws SQLException {
		// Get the connections for the specified user...
		Collection conns = (Collection) _userToConnectionsMap.get(username);
		// If none found, then NO connections are available...
		if (conns == null || conns.isEmpty()) {
			return null;
		}
		// ... otherwise, check to see if any of the connections are available...
		boolean isAvailable = false;
		final Iterator iterator = conns.iterator();
		PooledConnection pconn = null;
		while (iterator.hasNext()) {
			pconn = (PooledConnection) iterator.next();

			return pconn.getConnection();
			// Is this connection available? ignore this check for this sample test

		}
		return null;
	}

	public static void main(String[] args) throws SQLException, ClassNotFoundException, InterruptedException {

		Logger logger = Logger.getLogger("com.microsoft.sqlserver.jdbc.Statement");
		logger.setLevel(Level.FINER);
		Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");

		SQLConnectionPool pool = new SQLConnectionPool();

		try {

			Connection conn = pool.getConnection("<>", "<>");
			try (Statement stmt = conn.createStatement();) {

				String dropSql = "if exists (select * from dbo.sysobjects where id = object_id(N'[dbo].[" + tableName
						+ "]') and OBJECTPROPERTY(id, N'IsUserTable') = 1) DROP TABLE [" + tableName + "]";
				stmt.execute(dropSql);

				String createSql = "create table " + tableName + " (c1 int, c2 varchar(20))";
				stmt.execute(createSql);
			}
			for (int j = 0; j < 5; j++) {

				conn = pool.getConnection("<>", "<>");
				conn.setAutoCommit(false);
				try (PreparedStatement pstmt = conn.prepareStatement("insert into " + tableName + " values (?, ?)");) {
					for (int i = 0; i < 3; i++) {
						pstmt.setInt(1, i);
						pstmt.setString(2, "test" + i);
						pstmt.addBatch();
					}

					long start = System.currentTimeMillis();
					pstmt.executeBatch();
					conn.commit();
					long end = System.currentTimeMillis();

					System.out.println("Finished. Time taken : " + (end - start) + " milliseconds.");
				}
				conn.close();
			}
		} finally {
			// nothing for this test
		}

	}
}

This program results in incorrect entries being inserted to the test table with last item of the batch being repeated and the first item of the batch missed out.
Expected behavior
Caching and connection pool shouldn't lead to incorrect insert to the table. Should see no duplicates and no missing rows.
Actual behavior
At the end of the stand alone program run, could see there are (2*j) - 1 = 9 rows instead of 5 rows for column c1 : 2 and (j-1) = 4 entries missing for column c1 : 0

Error message/stack trace
With FINER logging enabled for SQL driver , could see the following exception : SQLException: com.microsoft.sqlserver.jdbc.SQLServerException: Could not find prepared statement with handle

Any other details that can be helpful
This issue does not happen when I comment the call to sp_reset_connection in SQLServerPooledConnection.java

JDBC trace logs
SQL Profiler data

hi @PriyadharshiniP
thanks for the repro we can see the issue will be implementing a fix for this in the upcoming preview.

Thanks for looking into this.

Additionally, it appears that the following modifications to the SQLServerPreparedStatement : by resetting the batchParam value to fetch the metadata of first row(0) instead of the last row seems to prevent duplicates from happening in this case.
But not entirely sure of the consequences.
Please have a look as it may help you fix this issue :

I'm also not clear why, when we re-use the same connection from the pool, a Could not find prepared statement with handle exception is thrown. I would really appreciate any clarifications to assist me better comprehend statement pooling.

I believe this is happening cos the cache is messed up. If I clear the preparedStatementHandleCache in resetPooledConnection() then this works. I'm just doing some testing now.

I appreciate you committing the fix.
It appears to prevent duplicates from happening by creating a new prepared statement handler for every usage of same connection from the connection pool. But I don't fully understand this fix.
I am aware that this statement pooling is per-connection and not server-wide.
But the prepared statement handler should be retained for re-using the connection having the same user name and password from pool : SQLServerPooledConnection , is it not ?
If statement caching was implemented and a connection pool was in use, it seems to be a waste in both client and server side and no real performance improvements with statement cache?
Please clarify.

Statement pooling is per connection, yes, but sp_reset_connection (resetPooledConnection) clears any prepared statements in the session on the server (among many other cleanup tasks). That's why you see the Could not find prepared statement with handle error. Clearing the cache would keep the driver from re-using invalid prepared statement handles. But I'd be worried that the driver is executing some of the batch or consuming/flushing results on the wrong side of the pool use boundary. ExecuteBatch() should be executing and consuming all the results from the server immediately and I wonder if it's leaving some hanging until the next statement execution (which is happening after the connection goes back into and then comes out of the pool again).

Thanks for clarifying that the statement pooling is per connection and cannot be shared even when the connection is used , goes back and is re-used from pool.
It appears that the un prepare call is not being initiated explicitly, and presumably, there can still be a performance improvement if statement pooling and pooling are enabled.
With the fix under this issue, it seems to have fixed the corrupt client side prepared statement cache by clearing it when the connection gets re-used from the pool.
But I am still not sure, if there can be some duplicates or entries from a batch mistakenly inserted due to any other scenarios where there is a corrupt  prepared statement cache or no server prepared handle with a retry. I cannot replicate such a scenario yet with 8179 and 586 exceptions. However it is in the exception handling, I think it may be possible.
The row in batchParam ought to have also been reset to the index of numBatchesPrepared rather than the last row if the retryBasedOnFailedReuseOfCachedHandle method returns true, right? I'm saying that there may still be instances where this exception stack trace occurs, and that the batch should be correctly retried.
Please advise.

yes we agree there may be other issues there we will take a closer look into this when we can, thanks

Hi @lilgreenbird
Thanks for fixing. May I know when a stable release (12.7.0) for this be available ?

The 12.7 preview release is available. The next stable release to include this will either be the 12.6.2 hotfix and/or the 12.8 GA release. We have not finalized a date for the hotfix, but the GA can be expected at the end of July (July 31).


Generate Tokio Runtime in ExecutionContext

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently a Tokio runtime is created outside of the control of DataFusion, and requires another user to specify a #[tokio::main] or a custom runtime.
The downside of this is that the config is outside of the control of DataFusion and depends on it being configured externally.
Describe the solution you'd like
Create a custom Runtime when creating the execution context.
Here we can also configure the worker thread count and max threads for the blocking thread pool based on a config or on number of CPU cores.
Possibly, for some usecases it would be nice to allow plugging in a runtime (e.g. when you want to share the runtime).
Describe alternatives you've considered
Additional context
Related to https://github.com/apache/arrow-datafusion/issues/924

FYI @alamb as this also has impact on IOx

If we go this approach, care must be taken to ensure any newly created executor does not interfere with one created with [tokio::main]
Here is an example of such a dedicated tokio executor that we made in IOx: https://github.com/influxdata/influxdb_iox/blob/main/query/src/exec/task.rs#L48-L65
I would be willing to contribute the DedicatedExecutor code back upstream to DataFusion if there is interest -- and we could add a configuration setting for thread counts, etc. I think the DedicatedExecutor also satisfies the usecase of "running DataFusion CPU-bound tasks on a different thread pool than IO bound tasks"

In case anyone is interested: https://docs.rs/tokio/1.12.0/tokio/#cpu-bound-tasks-and-blocking-code
Is upgraded to say that using a separate tokio executor is not a bad idea:

If your code is CPU-bound and you wish to limit the number of threads used to run it, you should use a separate thread pool dedicated to CPU bound tasks. For example, you could consider using the rayon library for CPU-bound tasks. It is also possible to create an extra Tokio runtime dedicated to CPU-bound tasks, but if you do this, you should be careful that the extra runtime runs only CPU-bound tasks, as IO-bound tasks on that runtime will behave poorly.


The client does not report SSE stream being closed by server

If the server closes a connection, the EventSource::state reports state is Open, and does not receive any error.
I have tried the Chrome developer console with the same server, and it reports an error and reopens the connection, which seems to be standard behaviour.
here's the minimal example I've used to test sse-client, closing connection upon finishing the stream is implied from axum inner works and confirmed by the maintainer here
use std::{env, sync::{Arc, atomic::AtomicI32}, time::Duration, convert::Infallible};

use axum::{Router, routing::get, response::sse::{Event, KeepAlive, Sse}};
use tokio;
use tracing::{info, warn};
use futures_util::stream::Stream;
use sse_client::EventSource;
use tower_http::cors::CorsLayer;

/// If a number is given as a parameter to the program, creates a client and expects this amount of messages to be received
#[tokio::main]
async fn main() {
  tracing_subscriber::fmt().init();
  let args: Vec<String> = env::args().collect();
  if args.len() > 1 {
      let amount = args[1].parse().unwrap();
      listen_sse(amount).await;
  } else {
      // create a server for serving SSE connections
      let app = Router::new().route("/sse", get(sse_handler)).layer(CorsLayer::permissive());
      axum::Server::bind(&"<IP_ADDRESS>:3000".parse().unwrap())
          .serve(app.into_make_service())
          .await
          .unwrap();
  }
}

/// Only send 2 events for each connection, then close the connection
async fn sse_handler() -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
    // A `Stream` that repeats an event every second
    let stream = async_stream::stream! {
        let mut interval = tokio::time::interval(Duration::from_secs(1));
        for counter in 1..3 {
            interval.tick().await;
            yield Ok(Event::default().data(format!("count = {counter}")));
        }
    };

    Sse::new(stream).keep_alive(KeepAlive::default())
}

/// Create a connection to SSE server, listen for a specified amount of events then close
async fn listen_sse(amount: i32) {
    let event_source = EventSource::new("http://localhost:3000/sse").unwrap();

    // This listener never gets executed
    event_source.add_event_listener("error", |error| warn!("Error {:?}", error));

    let counter = Arc::new(AtomicI32::new(1));
    let counter_clone = counter.clone();

    event_source.on_message(move |message| {
        info!("New message {:?}", message);
        counter.fetch_add(1, std::sync::atomic::Ordering::Release);
    });

    loop {
        tokio::time::sleep(Duration::from_millis(500)).await;
        //This keeps printing `stream state is Open` after the server has closed the connection
        info!("stream state is {:?}", event_source.state());
        if counter_clone.load(std::sync::atomic::Ordering::Acquire) > amount {
            break;
        }
    }

    event_source.close();
}

here's the Chrome console counterpart for the client side, I had to disable CSP to use this code, for that I've used this plugin:
const evtSource = new EventSource("http://localhost:3000/sse", { withCredentials: false });
evtSource.onmessage = (event) => { console.log(event); }
evtSource.onerror = (err) => { console.error("EventSource failed:", err); };

here's an example output for a version with more debug logs, using the sse-client:
2023-04-07T11:42:03.572910Z  INFO SERVER: axum_sse_playground: stream started
2023-04-07T11:42:03.574078Z  INFO SERVER: axum_sse_playground: sending "count = 1"
2023-04-07T11:42:03.574240Z  INFO client: axum_sse_playground: New message Event { id: "", type_: "message", data: "count = 1" }
2023-04-07T11:42:04.072047Z  INFO client: axum_sse_playground: stream state is Open
2023-04-07T11:42:04.574543Z  INFO client: axum_sse_playground: stream state is Open
2023-04-07T11:42:04.574581Z  INFO SERVER: axum_sse_playground: sending "count = 2"
2023-04-07T11:42:04.574865Z  INFO SERVER: axum_sse_playground: stream closed
2023-04-07T11:42:04.575359Z  INFO client: axum_sse_playground: New message Event { id: "", type_: "message", data: "count = 2" }
2023-04-07T11:42:05.076061Z  INFO client: axum_sse_playground: stream state is Open
2023-04-07T11:42:05.577693Z  INFO client: axum_sse_playground: stream state is Open
2023-04-07T11:42:06.079126Z  INFO client: axum_sse_playground: stream state is Open
2023-04-07T11:42:06.580785Z  INFO client: axum_sse_playground: stream state is Open
2023-04-07T11:42:07.081448Z  INFO client: axum_sse_playground: stream state is Open

using Chrome client:
2023-04-07T11:42:33.721796Z  INFO SERVER: axum_sse_playground: stream started
2023-04-07T11:42:33.723057Z  INFO SERVER: axum_sse_playground: sending "count = 1"
2023-04-07T11:42:34.723521Z  INFO SERVER: axum_sse_playground: sending "count = 2"
2023-04-07T11:42:34.723711Z  INFO SERVER: axum_sse_playground: stream closed
2023-04-07T11:42:37.736963Z  INFO SERVER: axum_sse_playground: stream started
2023-04-07T11:42:37.738235Z  INFO SERVER: axum_sse_playground: sending "count = 1"
2023-04-07T11:42:38.739625Z  INFO SERVER: axum_sse_playground: sending "count = 2"
2023-04-07T11:42:38.739805Z  INFO SERVER: axum_sse_playground: stream closed
2023-04-07T11:42:41.751556Z  INFO SERVER: axum_sse_playground: stream started
2023-04-07T11:42:41.752770Z  INFO SERVER: axum_sse_playground: sending "count = 1"
2023-04-07T11:42:42.752871Z  INFO SERVER: axum_sse_playground: sending "count = 2"
2023-04-07T11:42:42.752942Z  INFO SERVER: axum_sse_playground: stream closed

in browser:
MessageEvent {isTrusted: true, data: 'count = 1', origin: 'http://localhost:3000', lastEventId: '', source: null, …}
MessageEvent {isTrusted: true, data: 'count = 2', origin: 'http://localhost:3000', lastEventId: '', source: null, …}
EventSource failed: Event {isTrusted: true, type: 'error', target: EventSource, currentTarget: EventSource, eventPhase: 2, …}
MessageEvent {isTrusted: true, data: 'count = 1', origin: 'http://localhost:3000', lastEventId: '', source: null, …}
MessageEvent {isTrusted: true, data: 'count = 2', origin: 'http://localhost:3000', lastEventId: '', source: null, …}
EventSource failed: Event {isTrusted: true, type: 'error', target: EventSource, currentTarget: EventSource, eventPhase: 2, …}
MessageEvent {isTrusted: true, data: 'count = 1', origin: 'http://localhost:3000', lastEventId: '', source: null, …}
MessageEvent {isTrusted: true, data: 'count = 2', origin: 'http://localhost:3000', lastEventId: '', source: null, …}
EventSource failed: Event {isTrusted: true, type: 'error', target: EventSource, currentTarget: EventSource, eventPhase: 2, …}

Hello @sukhmel . Thanks for reporting this and for the very detailed example. I'll look into this later this week.


Is densenet (as the teacher) compatible with your method?

Hi there,
Since densenet, not like resnet, has many cross-block connections, I am wondering whether it, as the teacher, is compatible with your method?

Hi,
Personally, I think it is yes.
From my perspective to discuss, although there are no corresponding experiments about using DenseNet as the teacher, MGD helps the student to imitate the output features of the teacher. MGD does not consider how these output features are generated and what architecture creates these output features. So, MGD does not strongly depend on the architectures used in experiments.
Moreover, there is an experiment that ResNet distills ShuffleNet on ImageNet. The ShuffleNet apparently differs from the ResNet in the part of architecture because of the shuffle operation. Thus, I think, no matter the DenseNet will be the teacher or the student, it would work (maybe well like the case of ShuffleNet).

Thanks!

Hi,
Personally, I think it is yes.
From my perspective to discuss, although there are no corresponding experiments about using DenseNet as the teacher, it is to see that MGD helps the student to imitate the output features of the teacher. MGD does not consider how these output features are generated and what architecture creates these output features. So, MGD does not strongly depend on the architectures used in experiments.
Moreover, there is an experiment that ResNet distills ShuffleNet on ImageNet. The ShuffleNet apparently differs from the ResNet in the part of architecture because of the shuffle operation. Thus, I think, no matter the DenseNet will be the teacher or the student, it would work (maybe well like the case of ShuffleNet).

Thanks!


Fix for #308 write using specified dateFormat

Added support for writing to file with dateFormat that is specified in the options.  Added some basic tests for it.
I'm continuing to use SimpleDateFormat since that is what is used for reading. I would prefer joda's DateTimeFormat, but that can be a separate issue.
It's too bad CSVFormat does not allow specification of a dateFormat as it does for the null value. That would make this much easier.

Current coverage is 86.63%

Merging #310 into master will decrease coverage by -0.03% as of 3aa2e39

@@            master    #310   diff @@
======================================
  Files           12      12
  Stmts          540     546     +6
  Branches       160     161     +1
  Methods          0       0
======================================
+ Hit            468     473     +5
  Partial          0       0
- Missed          72      73     +1

Review entire Coverage Diff as of 3aa2e39

Powered by Codecov. Updated on successful CI builds.

Thanks for submitting this. I am going to review it. Meanwhile would you please take a look at the test failures?

When I compile and run locally it works, but I am using spark 1.6. Not sure what the build server is using. It complains that write is not a member on DataFrame, but it is in 1.6,

@barrybecker4 For me, I usually run this tests below:
sbt -Dspark.testVersion=1.3.0 test &&\
sbt -Dspark.testVersion=1.4.1 test &&\
sbt -Dspark.testVersion=1.6.0 test &&\
sbt scalastyle && \
sbt "test:scalastyle"

You can check why it fails in travis-ci.
It looks the reasons are below:
[error] /home/travis/build/databricks/spark-csv/src/main/scala/com/databricks/spark/csv/package.scala:170:0: Whitespace at end of line
[error] /home/travis/build/databricks/spark-csv/src/main/scala/com/databricks/spark/csv/package.scala:103: File line length exceeds 100 characters

I fixed the two style errors, and all those tests pass except for the last sbt "test:scalastyle" one.
It reports:

but does not say on what line, so I don't know how to fix it. Its also not in a file that I modified.

@barrybecker4 It looks your tests are using DataFrame.write() function which (AFAIK) does not exist in Spark 1.3.x. Maybe you should just follow the function usages in CSVSuite.scala.

I will look at using the alternative method to write. Is it really necessary to support all the way back to 1.3.x.?

I switched to using dates.saveAsCsvFile instead of dates.write so it will be compatible with spark 1.3.0.
I am seeing this stylecheck error in CsvSuite, but I did not change anything in that file.
spark\csv\CsvSuite.scala: expected start of definition, but was Token(VAL,val,2967,val)

@falaki I think it is ready to be reviewed.

@barrybecker4 thanks for fixing tests. I left a few comments. My main concern is the performance bottleneck inside the iterator. If you think there is risk of performance regression because of the additional type checking, would you change it to use schema information?

Are there any performance tests that indicate that it is significantly slower? I have not done any such tests yet. How much degradation is acceptable 0%, 5%, 10%?

We don't have any perf tests for spar-csv. Would you mind trying it manually and investigate?

@barrybecker4 I think we don't have to check if every value is Timestamp or Date type for every time since the data types of rows would be consistent as just @falaki said.
Isn't it obvious that checking only once would be faster than checking every value? I think we don't even need to test.

I cannot assert but this test will decrease the elapsed time by almost 20%
def time[R](block: => R): R = {
  val t0 = System.nanoTime()
  val result = block    // call-by-name
  val t1 = System.nanoTime()
  println("Elapsed time: " + (t1 - t0) + "ns")
  result
}

def test(v: Any) = {
  v match {
    case v: String => v
    case v: Int => v
    case _ => v
  }
}

val data = 0 to 10000000
time(data.foreach(v => test(v)))
time(data.foreach(v => v))

Are there any more changes that I need to make before this can be merged?

It's been a couple more months, and there have been no further comments. @HyukjinKwon  or @falaki, can this pull request be merged to master?

@barrybecker4 Looks good. I am merging this.
BTW: I am waiting for Spark 2.0 QA to conclude to make the final release of this package.

Unfortunately, I believe I just found a bug in the code that was merged. It fails if there is a null date in the date data. I will make a fix on my fork. Not sure what the process is. Do I reopen this issue or create a separate one?

I suggest you open a new issue and submit a new PR. This is just in master, since we have not published a release yet.


User database passwords are not kept in sync with user passwords

When we create the database_user, we set it up using database_password which depends on crypted_pasword(the hash of the user password):
https://github.com/CartoDB/cartodb/blob/master/app/models/user.rb#L1426
But when the user changes his password (crypted_password is changed), we don't update the PostgreSQL user password to match

There has been no activity on this issue for more several months. We are closing it. If you think this still needs to be addressed please open a new issue.


Fix integration tests

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 84.13%. Comparing base (94fb46b) to head (5149ad3).

Additional details and impacted files
@@            Coverage Diff            @@
##               main     #902   +/-   ##
=========================================
  Coverage     84.13%   84.13%
  Complexity     3577     3577
=========================================
  Files           372      372
  Lines         11696    11696
  Branches       1024     1024
=========================================
  Hits           9841     9841
  Misses         1400     1400
  Partials        455      455

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add new MANAGE_IDENTITIES permission

Add a new MANAGE_IDENTITIES permission to allow regular environment users to view the environment detail page.
Note that as part of this work, I've also tidied up the traits endpoints so that the FE can use those instead of the SDK endpoints as it currently does.

I think we need to do the same for edge identity and traits view set, right? Happy to chip in tho

Yes, good point. Should be pretty simple to add now that those viewsets are merged. I will do that now.


Fix for trailing / on route

This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
PR information

[ ] The title of the PR is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[ ] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[ ] If applicable, the PR references the bug/issue that it fixes.
[ ] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[ ] I have read the contribution guidelines.
[ ] My spec meets the review criteria:

[ ] The spec conforms to the Swagger 2.0 specification.
[ ] The spec follows the guidelines described in the Swagger checklist document.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-python

Can one of the admins verify this patch?

Automation for azure-sdk-for-go

Messed up syncing the fork, will recreate with a clean branch without conflicts

Automation for azure-sdk-for-java

Automation for azure-sdk-for-node


SERVER INVENTORY

NOM
SERVER INVENTORY
STORY
As a server park manager, I want to measure the impacts of my server park from a CSV inventory file
GUIDLINES

Provide an easy way to request the API for multiple servers
Provide an easy way to request the API from a CSV file
The retrieved impacts should be understood by humans and could provide insights to a server park manager

HELP
RESSOURCES

Exemple of CSV : https://docs.google.com/spreadsheets/d/1DqYgQnEDLQVQm5acMAhLgHLD8xXCG9BIrk-_Nv6jF3k/edit#gid=1494701657

Route [POST] /v1/server/

Example of server 1 - Complete server, die size are smart complete from components information, knowledge on electrical consumption :

{
  "model":
  {
    "manufacturer": "Dell",
    "name": "R740",
    "type": "rack",
    "year": 2020
  },
 {
    "configuration":
    {
        "cpu":
        {
            "units": 2,
            "core_units": 18,
            "family": "Skylake"
        },
        "ram":
        [
            {
                "units": 12,
                "capacity": 32,
                "manufacturer": "Samsung"
            }
        ],
        "disk":
        [
            {
                "units": 1,
                "type": "ssd",
                "capacity": 400,
                "manufacturer": "Samsung"
            }
        ],
        "power_supply":
        {
            "units": 2
        }
    }
},
  "usage": {
    "hours_use_time": 1,
    "days_use_time": 0,
    "years_use_time":  0
    "usage_location": "FRA",
    "hours_electrical_consumption": 510
  }
}

Example de server 2 - Complete server containing die size, electrical consumption is retrieved from workload object :

{
  "model":
  {
    "manufacturer": "Dell",
    "name": "R740",
    "type": "rack",
    "year": 2020
  },
  "configuration":
  {
    "cpu":
    {
      "units": 2,
      "core_units": 24,
      "die_size_per_core": 0.245
    },
    "ram":
    [
      {
        "units": 12,
        "capacity": 32,
        "density": 1.79
      }
    ],
    "disk":
    [
      {
        "units": 1,
        "type": "ssd",
        "capacity": 400,
        "density": 50.6
      }
    ],
    "power_supply":
    {
      "units": 2,
      "unit_weight": 2.99
    }
  },
  "usage": {
    "max_power": 510,
    "year_use_time": 3,
    "workload": {
      "100": {
        "time": 0.15,
        "power": 1.0
      },
      "50": {
        "time": 0.55,
        "power": 0.7235
      },
      "10": {
        "time": 0.2,
        "power": 0.5118
      },
      "idle": {
        "time": 0.1,
        "power": 0.3941
      }
    }
  }
}

ATTENTION
Need to provide a server inventory


[1.10.2] Client crash (JEI Int.)

JEI: jei_1.10.2-<IP_ADDRESS>5.jar
NEI: NotEnoughItems-1.10.2-<IP_ADDRESS>-universal.jar
Happends when I opened my inventory (creative).
If you need more information/list of mods, say so.
Stacktrace. https://gist.github.com/DeadSix27/b52379b2687d0aca9b8d5fcc9ea4d8e1

I am also getting this crash, fresh world as soon as the inventory is opened, chest, E keypress or any inventory - bam crash.
Crash Report: https://paste.ee/p/M6knx
Mod List: https://modlister.com/user/SteelShot

@mezz did you change anything inside JEI that could cause this?

Seems to be fixed now, using:
JEI: jei_1.10.2-<IP_ADDRESS>6.jar
NEI: NotEnoughItems-1.10.2-<IP_ADDRESS>-universal.jar
Whatever was in JEI 405
(Probably:

Current release <IP_ADDRESS>6
Fix #727 Recipes issues in previous commit

)
Fixed this crash.

That did not seem to fix anything, it still says that the build version is 405 in the crash log even to i just re downloaded it
https://paste.ee/p/9RWMB

Reverted 1 version back on NEI, does seem to fix my crash issues, will investigate more

Ok, this is really weird I removed some mods, Base Matals, Modern Metals MMO Ore Spawn and WalpaperCraft and Flat colored blocks, and it seems to work fine with the latest NEI version.
I guess this can be closed


neon-node-agent not starting due to port conflict

This happens occasionally (maybe 1% of cluster setups).  You'll find the details here:
https://github.com/nforgeio/neonKUBE/issues/1819#issuecomment-1632730593
This may explain the high neon-node-agent CPU utilization I was seeing a week or two ago while we were investigating high control-plane node CPU usage.
Unfortunately, neon-node-agent is not logging a stack trace.

Observations:

It looks like the neon-node-agent operator is hardcoded to listen on port 11006.  This may be a bad idea because the service is running in host network mode meaning the operator will be competing for this port with other processes running on the node, perhaps including new neon-node-operator instances being spun up for an upgrade, etc.  It's also possible that some completely unrelated process on the node may claim port 11006 before neon-node-agent does because we currently configure the Ubuntu ephemeral port range to 9000-65535.


Update format for bash and powershell

Fix format on bash and powershell

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: Jirapong sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

@Jirapong : Thanks for your contribution! The author, @rothja, has been notified to review your proposed change.

@Jirapong Thank you! This is a good correction. #sign-off


Impossible to change active tab styles

There is no way to change active tab styling as classTabActive is not added to the class list. It can be done manually, but it's not obvious for users.

Probably we need something like a button we used in the tooltip plugin

@artf thank you!


[Compactor] decode postings: invalid size

The compactor in my setup never runs and the container keeps failing with the following error:
ts=2023-09-21T12:37:50.748211273Z caller=azure.go:256 level=debug msg="getting blob" blob=01GT5CVYZNYFZDK93EJB8F8EQY/no-compact-mark.json offset=0 length=0
ts=2023-09-21T12:37:50.750633975Z caller=compact.go:597 level=error msg="retriable error" err="filter metas: filter blocks marked for no compaction: context canceled"
ts=2023-09-21T12:37:50.750748875Z caller=main.go:161 level=error err="group 0@10861838680888856476: invalid result block /data/compact/0@10861838680888856476/01HAVVWBTKQGZBRGS2M4JGRGXK: get all postings: decode postings: invalid size\ncompaction\nmain.runCompact.func7\n\t/app/cmd/thanos/compact.go:427\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:481\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:480\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\ncritical error detected\nmain.runCompact.func8.1\n\t/app/cmd/thanos/compact.go:495\ngithub.com/thanos-io/thanos/pkg/runutil.Repeat\n\t/app/pkg/runutil/runutil.go:74\nmain.runCompact.func8\n\t/app/cmd/thanos/compact.go:480\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\ncompact command failed\nmain.main\n\t/app/cmd/thanos/main.go:161\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"

Tested on 0.32, 0.32.2 and 0.32.3 versions.
Prometheus version is v2.22.1

Any updates on this?

Is it happening only for v0.32.X Thanos compactor?
Looks like it is compacting raw blocks and I am not sure if we changed that this release

Actually we started with an older version and after a while we enabled some prometheus rules and noticed this alert.

So my guess it never worked.
I enabled --no-debug.halt-on-error to get an idea where it actually fails. And tried to update to 0.32.0, 0.32.1, 0.32.2 and 0.32.3 but no change.

get all postings: decode postings

From this looks like the block is corrupted. Not sure if this is something that can be repaired by the bucket repair tool

@yeya24 how can force the deletion of said corrupted block?

@ylazaar We have a bucket tool to mark this block as deletion

@yeya24 I have already tried that.
the odd thing is i cannot find these blobs in azure's storage.

@yeya24 I have already tried that. the odd thing is i cannot find these blobs in azure's storage.

I have the same issue, how did you fix it?


Create qr.js

Generates QR code

Hey @Saksham-Aggarwal2812, this is an invalid PR. Please go through our contribution guide before raising another PR.


addStream() seems not working

Hello
What I'm trying to achieve:
By default users connect to each other and can only use a text chat.
When a user clicks a "share video" button, his video stream is shared among other users.
But in fact, when a user clicks the "share video" button, his local video stream is added to a page, but is not shared among other users. Other users receive the "ONSTREAM" event, but the stream is not active.

Please check this jsfiddle

Create a room
Join a room
3.Try to share a video

What am I doing wrong? How to make it work?
Thanks in advance

thank you very much for the demo


Fix entire page reload when an edit is made to an SDB.

Make it so that only the necessary elements on the page reload when a secret is added, edited, or deleted from an SDB.

I made a RC tag / release of this merged with master and deployed it to dev for you to checkout @sdford @tlisonbee


main: add option to ignore rule cache

capa's rule caching is great but not obvious. This caused a huge headache when debugging #1897 as the problem code was skipped entirely when capa used its local rule cache. I suggest we add a command-line option like --no-rule-cache to make it easier to disable the cache for situations like this. Otherwise, debugging code related to rule parsing requires finding (via the --debug option) and deleting the rule cache between subsequent executions.

first off, i'm sorry that you were bitten by this! i can only imagine that was a bit annoying.
i'm a little hesitant that we should add a new cli argument for this, since (ideally) no capa user would ever provide the flag. the cache detects changes to rule content but not source code content. the flag would only be relevant to capa developers that change capa logic (such as rule parsing).
could we instead disable the cache when running from source (eg. when installed by pip install -e .) and/or when run with --debug? or, if in source mode, use a hash of the capa source to derive the cache key?

This also got me before so the idea is good. I agree with Willi that another CLI argument should be avoided (plus I don't think I necessarily would remember it anyway). So, some automatic handling like also inspecting the hash of rule-related files sounds good.

Maybe we could introduce a new envrionement variable (e.g. DISABLE_CACHE_RULE=1) instead of the CLI argument?
@williballenthin's suggestion is also good. We could modify compute_cache_identifier to compute the cache ID not only based on the capa version and rules content, but also by including the hash of the source files.
This way, whenever the capa source code changes, the cache identifier will be different, and the existing cache will be invalidated. A new cache will be created the next time cache_ruleset is called. The only caveat (i.e. performance downgrade) here could be that we have to read in the source files to compute their hash. What do you think? I can draft a PR to test this out.

I'm not sure how to compute the set of file names that are used as source code, and I'm hesitant about getting bogged down figuring that out. If it's easy, then I'm ok exploring this a bit more.
I wonder if there's some way to interact with the Python interpreter's cache (pyc files) and derive the info that way.
Or could we use git status of the source repository?? Maybe this is simplest.
Anyways, I'm not sure this is the behavior that I want, since I may edit capa source dozens of times per day, and I don't think I want a new cache for each one. Maybe we could print a big red warning when the situation is detected?

Basically for source code, I was thinking about focusing on the *.py files.
Here is an example:
import hashlib
from pathlib import Path

def compute_cache_identifier(rule_content: List[bytes]) -> CacheIdentifier:
    hash = hashlib.sha256()

    # note that this changes with each release,
    # so cache identifiers will never collide across releases.
    version = capa.version.__version__

    hash.update(version.encode("utf-8"))
    hash.update(b"\x00")

    # Add the hash of the source files
    source_dir = Path(__file__).parent.parent
    source_files = list(source_dir.rglob("*.py"))
    for source_file in source_files:
        with open(source_file, "rb") as f:
            source_content = f.read()
        hash.update(hashlib.sha256(source_content).digest())

    rule_hashes = sorted([hashlib.sha256(buf).hexdigest() for buf in rule_content])
    for rule_hash in rule_hashes:
        hash.update(rule_hash.encode("ascii"))
        hash.update(b"\x00")

    return hash.hexdigest()

I believe this will introduce unnecessary overhead each time a user edits a file and re-runs capa, it will be noticable.

Or could we use git status of the source repository?? Maybe this is simplest.

git sounds like a good way to track changes, just unsure about how practical it is.

Anyways, I'm not sure this is the behavior that I want, since I may edit capa source dozens of times per day, and I don't think I want a new cache for each one. Maybe we could print a big red warning when the situation is detected?

We can. We just need to compute the hash using one of the aforementioned methods and alert. Users can then choose to ignore the warning, and generate the cache on-demand when needed.

git sounds like a good way to track changes, just unsure about how practical it is.

I understand the case we're trying to handle is that devs change source code in a way that invalidates the rules cache and it confuses them. So we can assume that this scenario involves a dev, and therefore git is present. And furthermore, we can rely on git to report the files that have been modified and only hash those ones.
This avoids the problem of inadvertently including irrelevant files in the hash.

See https://github.com/mandiant/capa-rules/blob/master/.github/scripts/create_releases.py for an example usage of git in one of our scripts.

I find this command to be suitable to our need:
git ls-files --deleted --modified --exclude-standard --full-name --deduplicate -v
R removed.txt                                   <- file was removed (rm removed.txt)
R renamed.txt                                   <- file was removed (mv tracked.txt renamed.txt)
C capa/rules/cache.py                           <- file was modified (vim cache.py)

Then we can filter out the deletions (marked as R). This will leave us with tracked, and modified files only.

Looks great!
We'll also want to incorporate the git commit hash.
This is shaping up well.

We'll also want to incorporate the git commit hash.

Yeah, this should help track committed changes which may affect rules / the cache.

As another alternative, can we compare the timestamps of capa/rules/cache.py vs. the most recent cache and print out a warning that this may result in unexpected behavior. We should keep this simple and little intrusive.

Regardless of the solution discussed after my initial message it appears that we'll still need to introduce a CLI argument, environment variable, etc. to control when the solution is executed. Otherwise, we'll be introducing overhead to all future invocations of capa just to handle a small use case where a developer may not want the cache to confuse their development?

Otherwise, we'll be introducing overhead to all future invocations of capa just to handle a small use case where a developer may not want the cache to confuse their development?

I think we can guess that we might be running in a dev environment very quickly:

not PyInstaller, which is by far the most common way to invoke capa, and
inspecting capa.main.__file__ doesn't contain site-packages, which should be very quick. (this might take some real world testing but i think the idea will work)

If these pass, we can then look for the .git directory (slightly slower) and then do the strategies already discussed (which will be fairly slow, but still only like 0.25s or so).
Therefore, I think it may still be possible to enable this for all runs, assuming we order the checks correctly.

Please note that the auto-cache generation approach will leave users will a lot of cache files in the cache dir.


tag v0.5.1

@JuliaRegistrator register branch=release/v0.5

Registration pull request created: JuliaRegistries/General/79467
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.5.1 -m "<description of version>" f47c8a086ab2eb37f2e9e634a5e26362ce408384
git push origin v0.5.1


INTER-3958: Format ppcp express pay buttons to match other express pay buttons in templates

Overview
Adjustments to the PPCP buttons to match the existing express pay buttons.

You know what would really spruce this PR up? Screenshots.

You know what would really spruce this PR up? Screenshots.

Checkout the screenshots in this PR: https://github.com/bold-commerce/checkout-experience-templates-internal/pull/104


Fix building an onchain app not rendering

What changed? Why?
Typo in typescript example was returning the code snippet rather than rendering it.
Notes to reviewers
How has it been tested?

good


version 0.1.4 vs version.yml vs cmd/gdmd/main.go

I assume something was forgotten with 0.1.4, as code mentions 0.1.3
It should be automated.
So, I have one question, what is version.yml file, and how do you use it?

It's a config file needed for a version management tool I developing https://github.com/chocolacula/verbum
Now it works in CI and this is an example of a log:
Commits to analyse:

ci: fix actions and update README
Merge pull request #19 from ccoveille-forks/add-badge
documentation: add godoc badge
ci: add golangci-lint and verbum
documentation: add godoc badge
Merge pull request #11 from ccoveille-forks/refactoring
chore(code): refactoring
Merge pull request #[9](https://github.com/chocolacula/gdmd/actions/runs/12126611639/job/33809072618#step:5:10) from ccoveille-forks/fix-install
fix(documentation): installation instructions
chore(code): refactoring
Always returns an error, never panics...
fix(documentation): installation instructions
Merge pull request #[10](https://github.com/chocolacula/gdmd/actions/runs/12126611639/job/33809072618#step:5:11) from ccoveille-forks/typos
chore(typo): fix styles
chore(typo): fix styles
fix: make help message better

component to up: patch
version bumped to 0.1.4


feat(ast): add ContentEq trait.

Part of #5283

I'll submit a follow-up PR addressing these issues and simplifying the implementation.

#5468 narrows the scope of ContentEq as discussed above.


awesome-propel list

Hi!
Based on other awesome lists I have the idea of creating an awesome list for propel (current 2).
An early draft is here: https://github.com/spryker/awesome-propel
TODOS:

[ ] adding more behaviors and relevant parts (anyone can help out making PRs), those should be up to date and CI green.

We can discuss

[ ] moving it into propel org directly - would be more visible this way from the start (assigned to someone who has the auth to do this)
[ ] once complete and fully reviewed, we can submit a PR also to the main awesome list (assigned to me)

So https://github.com/spryker/propel-encryption-behavior has been released.
Are there also others by the community?
Do we want to get started on such a list? What is the general consensus here?
Everyone is invited to participate.


Description example typo: "Alternat" instead of "Alternate

v0.23.0-rc6
Typo in the example text in the description textbox.

The screenshot didn't upload, but this was in "Add service alias".

build from master - June 9th
Verified fixed


Keep getting Unable to Login error

Hi all,
I'm new to this toolkit, but I have managed to get the verification code from my mail. I use outlook to read the mail.
I'm getting the error:

  e	{"Unable to login"}	System.Exception {UltimateTeam.Toolkit.Exceptions.FutException}

 e.InnerException	{"Index was outside the bounds of the array."}	System.Exception {System.IndexOutOfRangeException}

on this code:
ITwoFactorCodeProvider provider = new ImapTwoFactorCodeProvider();
LoginResponse respons = await FC.LoginAsync(loginDetails, provider);
return respons;
Any suggestions? Thanks!

Try use the same solution used here :)
https://github.com/trydis/FIFA-Ultimate-Team-Toolkit/issues/149

Thanks for the tip, but I already check that one. My mail account doesn't support the Imap connection so I solved it differently.
Here is a snippet (I checked it while debugging, it get the the code from my outlook and returns it!):
public Task GetTwoFactorCodeAsync()
{
return Task.Run(async () =>
{
await Task.Delay(10000);
var regex = new Regex(@"\d{6}");
                for (var i = 0; i < 7; i++)
                {
                    Microsoft.Office.Interop.Outlook.Application myApp = new Microsoft.Office.Interop.Outlook.Application();
                    var ns = myApp.Session;
                    var folders = ns.Folders[1].Folders[2];

                    var myInbox = folders.Items.Restrict("[Unread]=true");
                    if (myInbox.Count > 0)
                    {
                        for (int m = myInbox.Count; m > 0; m--)
                        {
                            int item = m;

                            // Grab the Body
                            var mail = (Microsoft.Office.Interop.Outlook.MailItem)myInbox[item];
                            string body = mail.Body;
                            // Sender Name
                            string sender = mail.SenderName;
                            if (sender == "EA")
                            {
                                Match match = regex.Match(body);
                                if (match.Success)
                                {
                                    mail.UnRead = true;
                                    return match.Value;
                                }
                            }
                        }
                    }

                        await Task.Delay(5000);
                    }

                    throw new FutException("Unable to find the two-factor authentication code.");
                });
            }

e-mail account, log in again at each cycle.

What do you mean?

Ok I changed it to a gmail account so I can use imap now. (withoud the using statement)
Use the code like below, but keep getting the index out of bounds error
public Task GetTwoFactorCodeAsync()
{
return Task.Run(async () =>
{
var regex = new Regex(@"((?<=class=""h1"">)\d{6})");
        var client = new ImapClient(_hostName, _username, _password, AuthMethods.Login, _port, _useUseSsl);

            for (var i = 0; i < 5; i++)
            {
                var messages = client.SearchMessages(SearchCondition.Unseen().And(SearchCondition.From("EA")));
                foreach (var match in messages.Select(message => regex.Match(message.Value.Body)).Where(match => match.Success))
                {
                    return match.Value;
                }
                await Task.Delay(5000);
            }

        throw new FutException("Unable to find the two-factor authentication code.");
    });
}

I have a SO too. Loginrequests.cs Row 170

https://github.com/trydis/FIFA-Ultimate-Team-Toolkit/issues/137
Fixed the Issue for me!

Worked for me as well!


InitContainer - inside Step

I want to change the permission of the mounted volume in InitContainer (inside Tekton Step).
Can this be done? I'm getting an error when I try to use InitContainers within Step.
I'm using the latest version of Tekton Pipeline. Could anyone please provide more info. on whether InitContainer can be used inside Tekton Step? If not, what are the alternatives?
Thanks in advance!

Are you trying to add an initContainer inside your step? Can you not add a previous step that does the same thing that your initContainer does?
/kind question

@dibyom - Very sorry for the delay. Since, the tekton step runs as a kubernetes container , user doesn't have access to run certain initializations within the step. That's the reason planned to use InitContainers. Was looking for something similar to Initconainers in Tekton or how Initcontainers can be initiated in Kubernetes. Thanks!

@2020testuser you can chain multiple steps. Take a look at the tar example in this section. You can find many other examples with multiple steps in the doc. So, I think you can just add a new "InitContainer"-like step before your main step?

@chanseokoh - I got the application successfully tested with gradle jib. Thanks for all your help.
Now , I'm testing with plain gradle (to do time comparison).
I'm testing with plain gradle (4.5 image). What happens is after gradle build (using :docker task), I need to push the built image which is in server host to the Docker repository. I have a secret created for my docker config in my Kubernetes namespace. Since, gradle 4.5  image uses gradle user, when I use docker push command to push to the registry , I'm getting 'Permission denied error' as I have the docker push command in the 'script' section of the gradle image step.
Note: I need to use the application code 'as is' to test gradle and so unable to use other docker related gradle task that builds the image and pushes to the repository. So, using plain docker command to push the created image to the docker repo.
Sorry for a new topic here. Thought you may have better ideas as you know what I have been doing because of the issue background. Thanks !

Because docker push pushes an image stored in a local Docker daemon (local to the running container), I assume in an earlier step you did gradle jibDockerBuild to build and push an image to the Docker daemon, right? Then, how about creating an image tarball instead of building an image to a Docker daemon? That is,

Run gradle jibBuildTar. This will create an image as a tarball at <project root>/build/jib-image.tar. (You can configure the tarball name if you want.)
In the next step, push the image tarball using a simple CLI tool like crane. Check the crane push doc. Note crane can get registry credentials from ~/.docker/config.json.

BTW, doing gradle jib is immensely more efficient (in terms of saving time, storage, and bandwidth) than doing gradle jibDockerBuild|jibBuildTar and then pushing the image later, because gradle jib can push only those bytes that are missing the target registry.

@chanseokoh - The Jib works fine and I will be using it.
I'm doing a comparison ( as a report) on build time in Jib and plain Gradle (without Jib and with :docker task ) to showcase jib and make it as standard.
That's the reason testing with plain Gradle.
In plain gradle, the gradle image runs as  gradle user (sample code below). So,  after building the image, I want to push the image to the company's docker repo. using docker push command. I tried with service account that includes the docker config as secret, tried with docker login etc. and no luck. Am I missing anything? Wondering why the docker login is not working (eventhough it is a right/cleaner approach). Any ideas? Thanks!
Note: The same script works fine if I run as root user( which I can't use becox of RBAC/security).
image: <my orh docker repo>/gradle:4.5     workingDir: /workspace     script: |       #!/bin/sh          set -o errexit        gradle --no-build-cache --gradle-user-home=/workspace/.gradle --recompile-scripts serviceapi:docker -x test /       -Duser.home=/workspace       docker tag serviceapi:latest <my org docker repo/imagename:tag>       docker login <my org docker url> -u <id> -p <encrypted password>              docker push  <my org docker repo/imagename:tag>

Even before thinking about a secret account, docker config and what not, there is a fundamental issue if you want to use Docker.
Basically, using Docker requires root. It is one of the dreaded requirements that people dislike but have to live with. You do need to have elevated privileges to be able to do docker push. At least the Docker daemon should run as root. (Not 100% sure but the Docker CLI might run as a non-root user if the user is in the privileged docker OS group. However, the Docker daemon should run as root, and the Docker CLI needs root privileges to be able to connect and use the daemon.)
I heard there is something called "rootless Docker" (experimental), but I am not sure if it really works.

@chanseokoh - Thanks! Could you please let me know whether having a step (say, step 2) after the gradle step (say step 1)?  In that way , step 2 can use Kaniko (or some other mechanism like shared workspace)  to pull the built image from step 1 and send as parameter to step 2. gradle user is not needed in step2 but as you mentioned is root usre still needed?
I'll also try to do some testing .
Thanks!

you mentioned is root usre still needed?

 But perhaps, now I am thinking you may not have to run docker push yourself. Please read through.
Disclaimer: I am not a Tekton expert, and what I say here may be out of date or not ideal.
Before I talk about pushing a Docker image from Tekton, I'd like to talk about general file and information sharing between steps. I believe there are multiple ways to achieve this. Maybe the simplest is to use the shared /workspace filesystem. This example creates a file /workspace/FOO in one step and the next step reads that file. Another way (I think) would be to mount your own volume. Not sure if Workspaces is another related thing. I also saw defining some kind of explicit input-output resource relation between steps, but I am not so sure.
However, I think Tekton has some degree of built-in support for handling Docker image output. This example in the Tutorial explains a Task fetching source code from a GitHub repository and building a Docker image from it.

The image resource specifies the repository to which the image built by the Task will be pushed:

The [Image resource type](a Task could fetch source code from a GitHub repository and build a Docker image from it.) represents an image in a remote registry.

It is usually used as a Task output for Tasks that build images. This allows the same Tasks to be used to generically push to any registry.

Note the same tutorial [section](The image resource specifies the repository to which the image built by the Task will be pushed:) also showcases an example where it uses Kaniko.
Hope this helps.

@chanseokoh - Thanks Much for providing more than one option !  Will try out these options.

@2020testuser just realized my last comment had broken links. Fixed.

@chanseokoh - Thanks!

I'm going to close this issue as it seems the original question has been largely resolved. Feel free to reopen if there's more to discuss / do here.


Initial Plugin Implementation

Description
An initial implementation of an asdf plugin to install and manage kubectl-buildkit versions.
Motivation and Context
We would like to use asdf to install kubectl-buildkit.
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Usage examples
As this plugin has not yet been accepted into the asdf-plugins repository, you'll need to use the following form to add it (once this PR is merged):
asdf plugin add kubectl-buildkit https://github.com/ezcater/asdf-kubectl-buildkit.git

To test out this PR, you'll need to switch the plugin to this PR's branch:
asdf plugin update kubectl-buildkit jb/DEVEX-1692/install-kubectl-build

Afterwards, you can use these commands to use this plugin:
# Show all installable versions
asdf list-all kubectl-buildkit

# Install specific version
asdf install kubectl-buildkit latest

# Set a version globally (on your ~/.tool-versions file)
asdf global kubectl-buildkit latest

# Now kubectl-buildkit commands are available
kubectl-buildkit version --help

How Has This Been Tested?

[x] Local testing

[x] linux
[x] osx
[ ] windows

[ ] Not needed, feature very basic

Checklist:

[x] I have updated the documentation accordingly.
[x] I have added tests to cover my changes.

@mikowitz Check out the usage examples in the PR description. 😉

:shipit:


.sqlite file discussion

The vision tha I had for the .sqlite file in this repository was that it was an "easy" way for someone to poke around the template schema.  It's not supposed to be blank! It's supposed to hold a complete template database.
Now, whether it should go in the .gitignore is another issue which I cant quite wrap my head around right now but at least I started the conversation!

Just spitballing here:
If we want the .sqlite file to contain a complete template database, I don't think it should go in the gitignore. Our definition of what constitutes a "complete template database" is certainly liable to change over time, and I think we should include a version of the database here that reflects that.

The .gitignore needs to be modified to reflect this discussion.


BUG: Operator cleanup removes entire namesace

Describe the bug
When the operator is cleaned up, it removes the entire namespace including resources which are not managed by the operator.
To Reproduce
Steps to reproduce the behavior:

Apply operator to namespace with existing resources
Remove operator from namespace

Expected behavior
It should be expected that the operator would clean up the resources it manages: Config Map, Stateful Set, Persistent Volume Claims, Persistent Volumes, Service, Headless Service, etc.
All resources not managed by the operator should be left untouched.
Please complete the following information:

Kubernetes version

1.28.5 (eks)
1.32.0 (minikube)

Typesense version

0.25.0
0.25.2

Cloud provider or self-managed?

AWS (EKS)
self-managed localhost (minikube)

Additional context
This bug could potentially cause serious damage when applied to a working environment.

@psaxton Thanks for pointing it out, will release a fix soon.

This is Fixed.


Add support for pacman

Please consider adding support for pacman(Arch Linux package manager).

I am an Arch user now, and this is how I've been using here.

Set your sudo config to use http_proxy and https_proxy from the user running the sudo command.
OR
Use sudo -E for sudo. This preserves environment variables for sudo command used.

For references, refer https://stackoverflow.com/questions/8633461/how-to-keep-environment-variables-when-using-sudo


Are combined types (sections) possible?

In PyInstaller we have different "areas", a bugfix, enhancement, etc. can occur: the main app, "hooks", or the "bootloader". Also some changes may only apply to certain operating-systems. Does towncrier allow to combine framgent-types?
Example result:

Bootloader

The bootloader is now …
(Windows) Short Paths now work even f written right-to-left.

Hooks

Added hooks for summer, winter, snow.
(OS X) Fixed hook autumn

Answering myself: There is some "section" feature yet to-be documented, see #61. Reading the code it looks like each "section" (to be listed in the .toml-file) will be a sub-directory in the newfragments directory.


Automatic inclusion of sqlite3.dll for UWPs defeats the purpose of WinSqlite.

Currently if the rc2 nuget package is referenced from a UWP, it automatically includes a copy of sqlite3.dll in the AppX package. This pretty much defeats one of the primary purposes of WinSqlite, which is to reduce app package size and reduce the number of times sqlite3.dll is present in various apps.
One potential solution to this is to check for an MSBuild property (say DisableIncludeSqlite) and check if that's set in the targets file, and if so don't include it.
This would also enable developers to provide custom (perhaps newer) builds of sqlite3.dll if needed.

We're discussing possible solutions to this in #148

For now, you can remove it by adding the following to your *.csproj file.
<Target Name="RemoveSqlite" AfterTargets="ResolveNuGetPackageAssets">
  <ItemGroup>
    <ReferenceCopyLocalPaths Remove="%(ReferenceCopyLocalPaths.Identity)"
                             Condition="'%(Filename)%(Extension)' == 'sqlite3.dll' " />
  </ItemGroup>
</Target>

Unfortunately, .NET Native takes issue with this. (See #249)

Closing as this is covered by #148 and #249.


ML_pipelines: managing table reading

I am using Spark ML_pipelines to easily deploy operations that I have developed in Sparklyr in a production environment using SCALA. It is working pretty well, except for one part: it seems that when I read a table from Hive and then create a pipeline that applies operations to this table the pipeline will also save the table reading operation and thereby the name of the table. However I want the pipeline to be independent of this.
Here is a reproducible example:
Sparklyr part:
sc = spark2_context(memory = "4G")

iris <- copy_to(sc, iris, overwrite=TRUE)

spark_write_table(iris, "base.iris")
spark_write_table(iris, "base.iris2")
df1 <- tbl(sc, "base.iris")

df2 <- df1 %>%
  mutate(foo = 5)

pipeline <- ml_pipeline(sc) %>%
  ft_dplyr_transformer(df2) %>%
  ml_fit(df1)

ml_save(pipeline,
        paste0(save_pipeline_path, "test_pipeline_reading_from_table"),
        overwrite = TRUE)

df2 <- pipeline %>% ml_transform(df1)

dbSendQuery(sc, "drop table base.iris")

SCALApart:
import org.apache.spark.ml.PipelineModel
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

val df1 = spark.sql("select * from base.iris2")

val pipeline = PipelineModel.load(pipeline_path + "/test_pipeline_reading_from_table")
val df2 = pipeline.transform(df1)

I get this error:
org.apache.spark.sql.AnalysisException: Table or view not found: `base`.`iris`; line 2 pos 5;
'Project ['Sepal_Length, 'Sepal_Width, 'Petal_Length, 'Petal_Width, 'Species, 5.0 AS foo#110]
+- 'UnresolvedRelation `base`.`iris`

  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:82)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:78)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at scala.collection.immutable.List.foreach(List.scala:381)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:78)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:91)
  at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:52)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:67)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:637)
  at org.apache.spark.ml.feature.SQLTransformer.transformSchema(SQLTransformer.scala:86)
  at org.apache.spark.ml.PipelineModel$$anonfun$transformSchema$5.apply(Pipeline.scala:310)
  at org.apache.spark.ml.PipelineModel$$anonfun$transformSchema$5.apply(Pipeline.scala:310)
  at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
  at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
  at scala.collection.mutable.ArrayOps$ofRef.foldLeft(ArrayOps.scala:186)
  at org.apache.spark.ml.PipelineModel.transformSchema(Pipeline.scala:310)
  at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74)
  at org.apache.spark.ml.PipelineModel.transform(Pipeline.scala:304)
  ... 71 elided

I can see 2 solutions:

It seems that persisting dataframe would be a solution, but then I would need to find a way not to overload my memory, hence my question on unpersisting on SO

Passing the name of the table in Hive as a parameter of the pipeline, which I am trying to solve in this question on SO

Now, all of this being said, I might be missing something as I am only a beginner...

@vl2376 it seems the usage problem from your side.

It could not be found because I deleted it on purpose. My point is to know how I can make my pipeline independent from my reading operation.


Feature/support maghrib angle

Fixes #11

@phoenix-lab-io can you add tests based on this file: https://github.com/batoulapps/adhan-js/blob/master/Shared/Times/Tehran-Tehran.json
This is the test: https://github.com/iamriajul/adhan-dart/blob/9f1504e2d4fdb4a632f00e1b567806e796e2a590/test/src/prayer_times_test.dart#L170


NO_PROXY does not support CIDR notation. #8191

NO_PROXY does not support CIDR notation. #8191

Proposed changes

Unreleased project version (optional)

Related issues (optional)

Looks like you opened the pull request against your own repository instead of the docker repository; could you open a pull request against the docker documentation repository using this link? https://github.com/docker/docker.github.io/compare/master...ryujisnote:ryujisnote-noproxy-nocidr-1?expand=1

Thank you four your comment.
I created PR against.


Improve Stackage nightly workflow to automatically create an issue if it fails

Summary

Checklist

[ ] All definitions are documented with Haddock-style comments.
[ ] All exported definitions have @since annotations.
[ ] Code is formatted in line with the existing code.
[ ] The changelog has been updated.

Superseded by #63 where this behaviour is incorporated into mbg/actions/stack/nightly.


Bump jose from 2.0.5 to 2.0.6

Dependabot has already raised a PR  for this.
Please also read through this part of the volunteer wiki to get more information on how to complete dependency upgrade issues.
Approach

Clone the branch raised by dependabot to your local computer
Run up bloom-backend to ensure that this change in dependency has not affected the site
Run tests to ensure they're all passing
Commit any changes to the branch and push up
Comment on the ticket that this change has been verified and request reviews from the team (@swetha-charles or @eleanorreem)

I will pick this issue (:


To groups.py add share_group, unshare_group, list_shared_with, and a test

Coverage of the new code is good (measured by coverage run --include src/tiledb/cloud/groups.py -m pytest tests/test_groups.py). We lack tests of errors sent by the API server, but the code around that is implicitly trusted boilerplate.
Once this code is merged, I plan to wrap these up in functions in the new tiledb.cloud.asset module.

@JohnMoutafis @thetorpedodog yea or nay? Other suggestions?


Started a Rubix ML client

One of Rubix ML case studies is a sentiment analyzer. It is packed here as a local backend.
A trained model is included in the test cases.
This model is not very good with short pieces of text but it runs locally without any cloud fees.

@DrDub Dude, I totally didn't see this notification. I've got dependepot alerts cluttering my inbox from an old client's repo, apologies! I'll review this asap. I actually started noodling with a cloud-free implementation of Rubix ML but you appear to be further down the road on this than me. Thanks a ton for your contribution!

@DrDub merging to a feature branch to tweak a few things but overall great PR. Thank you so much for your contribution! Curious: how did you train the sentiment fixture?

Hi @christopherarter! The sentiment model comes from the Rubix ML example project: https://github.com/RubixML/Sentiment . It is trained on 25,000 movie reviews from the IMDB. It is not necessarily a great model but it is a model nevertheless.


feat(enum): initial support

Hey, @michaelvanstraten amazing crate, saved me from writing 80 lines of code for each type I want to use with redis.
I've manged to support enums with no fields. However, when an enum has fields I simply just paniced.
I'm not sure how to support for enum with unamed and named fields, we can only have string type at the end. Let me know if you have any ideas.

Foremost, thank you for the praises and contributing to the Codebase.
The PR looks good, but I will try to take a crack at the unnamed fields, maybe ill find something.

No problem, thank you for creating such a Codebase.
I was thinking maybe I could support customizing the casing of the the variants (i.e. capitalized/snake_case/uppercase, with the default being capitalized) then we should just merge, if later you decided on supporting unnamed fields, a different PR would be better.

@michaelvanstraten okay I'm ready for review. After that we should merge, I feel I'm getting out of the scope of the PR 😆

@michaelvanstraten okay I'm ready for review. After that we should merge, I feel I'm getting out of the scope of the PR 😆

Haha, maybe we should focus on one thing at a time.
I'm still not home, so I will hopefully get it merged by this evening.
I am working on a different approach to serializing data into one big Redis hash, perhaps you could share your opinion on that. I'll write something up as son as I am home.

@kkharji sorry for the long wait, I am having a very busy week.
The Problem with the current implementation is that it relies too much on the usage of a single Redis hash.
To effectively store more complex data structures on Redis, we have to rely on other Data types. But to do this, the FromRedisValue and ToRedisArgs traits are too inflexible. For Example, we would like to push this struct to the database:
struct User {
    id: u32,
    name: String,
    follows: Vec<u32>
}

The id as well as the name can be easily represented as a single key value pair on the hash, but the follows field?
A Vector does not have a clear and simple conversion to a pair of key and value, you have to rely on a Redis List or set. The current implementation is pretty wacky and isn't really flexible to use in future continuation.
Now you have the problem that you have to send a Pipeline to push the struct to the database and send a Pipeline to get the struct from the database. This is in conflict with the FromRedisValue and the ToRedisArgs trait implementation.
And there is a good reason why the two traits are implemented this way, and changing the current implementation would not be logical just to allow this to work.
So to continue working on this crate, I would have to move away from FromRedisValue and ToRedisArgs.
The hole purpose I developed this is to integrate seamlessly into the redis-rs ecosystem, so moving away from  FromRedisValue and ToRedisArgs would defeat that purpose. This is my dilemma and the reason I have been putting it off to work on this crate.

In the new crate I have been working on, I am serializing structs and enums in a hole different way.
Fields which value will not be serializable to a single value would each create a nested key and set their value recursively.
For Example, the before mentioned user struct would become a Redis hash with the ident of "user1", and a Redis list with the ident of "user1::follows".
To set this Struct on the Database, a pipeline like this would be generated:
redis::pipe()
    .cmd("HSET").arg("user1").arg("id").arg(42).arg("name").arg("Neo")
    .cmd("DEL").arg("user1::follows")
    .cmd("LPUSH").arg("user1::follows").arg(34).arg(12).arg(25);

🤔 it's indeed a hole new and unconventional way. I see what you are trying to achieve with the new crate "btw invite me if you are going to open source it anyway soon".
In this way, you can't use redis-rs commands and you have to provide you're own commands for getting and saving a data type, let alone other functionalities.
I understand the value of this approach. It sounds like a fun and interesting problem to solve.
I suggestion due to the focus of this crate at playing well with redis-rs and that most users are okay with storing complex data type as json (integrating serde). We can do the following with slight changes in the macro generator:

lists are index by number as mentioned in #6: follows.0, follows.1
struct field that implements FromRedisValue, ToRedisArgs and RedisComplexType, e.g. struct AccountInfo { balance: u32, creation_date: String, } get stored as info.balance, info.creationDate
If struct doesn't implement the above trait, and implement serde, we can just use serde for that key.

If you have discord, you can contact me at tami5#1560

🤔 it's indeed a hole new and unconventional way. I see what you are trying to achieve with the new crate "btw invite me if you are going to open source it anyway soon".
In this way, you can't use redis-rs commands and you have to provide you're own commands for getting and saving a data type, let alone other functionalities.
I understand the value of this approach. It sounds like a fun and interesting problem to solve.
I suggestion due to the focus of this crate at playing well with redis-rs and that most users are okay with storing complex data type as json (integrating serde). We can do the following with slight changes in the macro generator:
* lists are index by number as mentioned in [feat: support rename_all and rename struct fields #6](https://github.com/michaelvanstraten/redis-derive/pull/6): `follows.0, follows.1`

* struct field that implements `FromRedisValue`, `ToRedisArgs` and `RedisComplexType`, e.g. struct AccountInfo { balance: u32, creation_date: String, } get stored as `info.balance, info.creationDate`

* If struct doesn't implement the above trait, and implement serde, we can just use serde for that key.

If you have discord, you can contact me at tami5#1560

I have sent you a friend request on discord.


fiexed typo in ramp method documentation

fixes #467
removed hard coded (optional)
because writing [v2] is understood by yuidocs as optional parameter and it automatically writes there '(Optional)` for us !

@therewasaguy  please review this pr!


coprv2: config for plugin directory

What problem does this PR solve?
Issue Number: #9747
Problem Summary:
Adds a new configuration section with one configuration option to TiKV's configuration.
[coprocessor-v2]
coprocessor-plugin-directory = "coprocessors"

The config is then passed to the new coprocessor endpoint where it will be registered for hot-reloading.
What is changed and how it works?
Proposal: RFC
What's Changed:
Instead of adding every plugin individually to TiKV's config, we just use one "plugin directory".
Related changes

based on PR #9925

Check List
Tests

I'm not entirely sure how to test this. Also I'm not quite sure how to test this manually :see_no_evil:

Release note

No release note

looking forward to it.


errors when specifying a config file

on ubuntu 11.04 (x64), python version 2.7.1+
when specifying a config file:
bucky /path/to/bucky_conf.py
I get the error:
"No handlers could be found for logger "bucky.main""
edit:
sorry, neglected to mention the version of bucky I'm using, which is the latest master

there is a syntax error in your config file.


"up account login throws Error: Bad Request if already logged in

Prerequisites

[x] I am running the latest version. (up upgrade)
[x] I searched to see if the issue already exists.
[x] I inspected the verbose debug output with the -v, --verbose flag.
[x] Are you an Up Pro subscriber?

Description
I expected to see that I was already logged in when running up account login not Error: Bad Request
Steps to Reproduce

Be logged in to up already
Run up account login
See Error: Bad Request

duplicate of #474


Request: Offline maps

Hey! Awesome job on this module - got it working "right out of the box" :)
Just wondering if you have thought about including an ability to use offline maps?

My wondering: how can I add/remove annotation?


Event Handlers

When I'm providing either of one Single Click or Double click then its working as expected. But when I'm providing both events, only single click event is getting fired.
elementDblclick: function (e) {
                              $log.info("Double Click");
                          },
elementClick: function(e) {
                              $log.info("Single Click");
                          }

My scenario is on click it should call one function with the chart data and double click it should go to  a url with chart data.

Maybe if you subscribe on elementClick event and then do double click, the chart tries to recognize it as a two single click events that have been done quickly. So before elementDblclick event, you always do a couple of elementClick events. My advice is try to think of another event scenario in your case.

Thanks for reply. I'll try to modify my scenario.
Is there any way I can detect Click event Outside the Bars in case of Multi Bar Charts?